PyPSA / PyPSA/linopy

spec: grouped operators crash on string labels under pandas 3.0 (ArrowStringArray)

Open
#924 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug math-spec
Dominant language
Python
Stars
257
Forks
87
Avg merge
1d 3h
Merged PRs (30d)
29

Description

[!NOTE]
This bug report was drafted by an AI agent from a reproduced failure. The
reproduction, traceback and cause below are verbatim from the run.

Building a spec model that groups over a string dimension crashes under
pandas 3.0. The grouped operators are affected: sum(by=…), shift(…, by=…),
sum_back(…, by=…) and at.

pandas 3.0 defaults string data to the Arrow-backed string dtype
(future.infer_string = True), so a lookup whose values are strings arrives as
an ArrowStringArray. linopy/spec/operators.py::_grouped then calls
DataArray.sel(...) on that lookup, and the installed xarray refuses to index
an ArrowStringArray (as_indexable raises TypeError: Invalid array type).

Environment
  • linopy: spec-builder branch
  • pandas 3.0.5, xarray 2026.7.0, numpy 2.4.6, Python 3.12.3
Minimal reproduction
import pandas as pd
import linopy

linopy.options["semantics"] = "v1"
spec = {
    "dimensions": {"t": {"dtype": "str"}, "g": {"dtype": "str"}},
    "lookups": {"grp": {"over": "t", "into": "g"}},
    "parameters": {"v": {"dims": ["t"]}},
    "variables": {
        "x": {"foreach": ["t"], "bounds": {"lower": 0, "upper": 100}},
        "y": {"foreach": ["t"], "bounds": {"lower": -100, "upper": 100}},
    },
    "constraints": {
        "fix": {"foreach": ["t"], "expression": "x == v"},
        "link": {"foreach": ["t"],
                 "expression": "y == shift(x, over=t, offset=1, edge=0, by=grp)"},
    },
    "objective": {"sense": "minimize", "expression": "sum(x)"},
}
data = {
    "t": ["t0", "t1", "t2"], "g": ["g0", "g1"],
    "grp": pd.Series(["g0", "g0", "g1"], index=["t0", "t1", "t2"]),
    "v": pd.Series([1.0, 2.0, 3.0], index=["t0", "t1", "t2"]),
}
linopy.Model.from_spec(spec, data)   # -> TypeError
Cause, confirmed

Adding pd.set_option("future.infer_string", False) before the build makes the
same model build and solve. The trigger is the pandas-3.0 string default
producing an ArrowStringArray for the lookup's values.

Suggested fix

In _grouped (linopy/spec/operators.py:263), take the lookup values off the
Arrow-backed dtype before indexing — build keys from a plain object/numpy
view of the values, or avoid .sel here — so string groups index the same way
int and object groups already do.

Full traceback
Traceback (most recent call last):
  File "repro.py", line 31, in <module>
    linopy.Model.from_spec(spec, data)
  File ".../linopy/model.py", line 510, in from_spec
    return cls(**model_kwargs).add_spec(spec, sources, retain=retain)
  File ".../linopy/model.py", line 494, in add_spec
    self._spec = attach(self, spec, sources, retain)
  File ".../linopy/spec/accessor.py", line 59, in attach
    build(model, bound)
  File ".../linopy/spec/builder.py", line 61, in build
    _constraints(ctx)
  File ".../linopy/spec/builder.py", line 134, in _constraints
    lhs, rhs = evaluate(row.lhs, ctx), evaluate(row.rhs, ctx)
  File ".../linopy/spec/builder.py", line 225, in evaluate
    return operators.shift(
  File ".../linopy/spec/operators.py", line 128, in shift
    groups = _grouped(over, np.asarray(array.indexes[over]), by)
  File ".../linopy/spec/operators.py", line 263, in _grouped
    keys = np.asarray(groups.sel({over: labels}).values, dtype=object)
  File ".../xarray/core/dataarray.py", line 1729, in sel
    ds = self._to_temp_dataset().sel(...)
  File ".../xarray/core/dataset.py", line 3086, in sel
    result = self.isel(indexers=query_results.dim_indexers, drop=drop)
  File ".../xarray/core/dataset.py", line 2939, in isel
    var = var.isel(var_indexers)
  File ".../xarray/core/variable.py", line 1144, in isel
    return self[key]
  File ".../xarray/core/variable.py", line 829, in __getitem__
    indexable = as_indexable(self._data)
  File ".../xarray/core/indexing.py", line 1030, in as_indexable
    raise TypeError(f"Invalid array type: {type(array)}")
TypeError: Invalid array type: <class 'pandas.arrays.ArrowStringArray'>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in linopy/spec/operators.py at _grouped, reached from shift(..., by=...) in the supplied reproduction. Run the minimal pandas 3.0 example and inspect how lookup values become keys before DataArray.sel; done means grouped sum, shift, sum_back, and at work with ArrowStringArray-backed string labels without changing the existing int and object behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.