spec: grouped operators crash on string labels under pandas 3.0 (ArrowStringArray)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 257
- Forks
- 87
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 29
Description
[!NOTE]
This bug report was drafted by an AI agent from a reproduced failure. The
reproduction, traceback and cause below are verbatim from the run.
Building a spec model that groups over a string dimension crashes under
pandas 3.0. The grouped operators are affected: sum(by=…), shift(…, by=…),
sum_back(…, by=…) and at.
pandas 3.0 defaults string data to the Arrow-backed string dtype
(future.infer_string = True), so a lookup whose values are strings arrives as
an ArrowStringArray. linopy/spec/operators.py::_grouped then calls
DataArray.sel(...) on that lookup, and the installed xarray refuses to index
an ArrowStringArray (as_indexable raises TypeError: Invalid array type).
Environment
- linopy:
spec-builderbranch - pandas 3.0.5, xarray 2026.7.0, numpy 2.4.6, Python 3.12.3
Minimal reproduction
import pandas as pd
import linopy
linopy.options["semantics"] = "v1"
spec = {
"dimensions": {"t": {"dtype": "str"}, "g": {"dtype": "str"}},
"lookups": {"grp": {"over": "t", "into": "g"}},
"parameters": {"v": {"dims": ["t"]}},
"variables": {
"x": {"foreach": ["t"], "bounds": {"lower": 0, "upper": 100}},
"y": {"foreach": ["t"], "bounds": {"lower": -100, "upper": 100}},
},
"constraints": {
"fix": {"foreach": ["t"], "expression": "x == v"},
"link": {"foreach": ["t"],
"expression": "y == shift(x, over=t, offset=1, edge=0, by=grp)"},
},
"objective": {"sense": "minimize", "expression": "sum(x)"},
}
data = {
"t": ["t0", "t1", "t2"], "g": ["g0", "g1"],
"grp": pd.Series(["g0", "g0", "g1"], index=["t0", "t1", "t2"]),
"v": pd.Series([1.0, 2.0, 3.0], index=["t0", "t1", "t2"]),
}
linopy.Model.from_spec(spec, data) # -> TypeError
Cause, confirmed
Adding pd.set_option("future.infer_string", False) before the build makes the
same model build and solve. The trigger is the pandas-3.0 string default
producing an ArrowStringArray for the lookup's values.
Suggested fix
In _grouped (linopy/spec/operators.py:263), take the lookup values off the
Arrow-backed dtype before indexing — build keys from a plain object/numpy
view of the values, or avoid .sel here — so string groups index the same way
int and object groups already do.
Full traceback
Traceback (most recent call last):
File "repro.py", line 31, in <module>
linopy.Model.from_spec(spec, data)
File ".../linopy/model.py", line 510, in from_spec
return cls(**model_kwargs).add_spec(spec, sources, retain=retain)
File ".../linopy/model.py", line 494, in add_spec
self._spec = attach(self, spec, sources, retain)
File ".../linopy/spec/accessor.py", line 59, in attach
build(model, bound)
File ".../linopy/spec/builder.py", line 61, in build
_constraints(ctx)
File ".../linopy/spec/builder.py", line 134, in _constraints
lhs, rhs = evaluate(row.lhs, ctx), evaluate(row.rhs, ctx)
File ".../linopy/spec/builder.py", line 225, in evaluate
return operators.shift(
File ".../linopy/spec/operators.py", line 128, in shift
groups = _grouped(over, np.asarray(array.indexes[over]), by)
File ".../linopy/spec/operators.py", line 263, in _grouped
keys = np.asarray(groups.sel({over: labels}).values, dtype=object)
File ".../xarray/core/dataarray.py", line 1729, in sel
ds = self._to_temp_dataset().sel(...)
File ".../xarray/core/dataset.py", line 3086, in sel
result = self.isel(indexers=query_results.dim_indexers, drop=drop)
File ".../xarray/core/dataset.py", line 2939, in isel
var = var.isel(var_indexers)
File ".../xarray/core/variable.py", line 1144, in isel
return self[key]
File ".../xarray/core/variable.py", line 829, in __getitem__
indexable = as_indexable(self._data)
File ".../xarray/core/indexing.py", line 1030, in as_indexable
raise TypeError(f"Invalid array type: {type(array)}")
TypeError: Invalid array type: <class 'pandas.arrays.ArrowStringArray'>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in linopy/spec/operators.py at _grouped, reached from shift(..., by=...) in the supplied reproduction. Run the minimal pandas 3.0 example and inspect how lookup values become keys before DataArray.sel; done means grouped sum, shift, sum_back, and at work with ArrowStringArray-backed string labels without changing the existing int and object behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100