ArrowInvalid: offset overflow while concatenating arrays
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
Describe the bug
I want to split samples by their regions for a Xenium slide. I read the samples' area as polygon, but ArrowInvalid when I ran polygon_query. How to fix this? Thanks!
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[17], line 1
----> 1 sdata_crop = sd.polygon_query(
2 sdata_biopsy,
3 polygon=polygon,
4 target_coordinate_system='global',
5 )
File /opt/conda/lib/python3.10/functools.py:889, in singledispatch.<locals>.wrapper(*args, **kw)
885 if not args:
886 raise TypeError(f'{funcname} requires at least '
887 '1 positional argument')
--> 889 return dispatch(args[0].__class__)(*args, **kw)
File /opt/conda/lib/python3.10/site-packages/spatialdata/_core/query/spatial_query.py:831, in _(sdata, polygon, target_coordinate_system, filter_table, clip, shapes, points, images, labels)
829 for element_type in ["points", "images", "labels", "shapes"]:
830 elements = getattr(sdata, element_type)
--> 831 queried_elements = _dict_query_dispatcher(
832 elements,
833 polygon_query,
834 polygon=polygon,
835 target_coordinate_system=target_coordinate_system,
836 clip=clip,
837 )
838 new_elements[element_type] = queried_elements
840 tables = _get_filtered_or_unfiltered_tables(filter_table, new_elements, sdata)
File /opt/conda/lib/python3.10/site-packages/spatialdata/_core/query/spatial_query.py:422, in _dict_query_dispatcher(elements, query_function, **kwargs)
420 assert isinstance(d, dict)
421 if target_coordinate_system in d:
--> 422 result = query_function(element, **kwargs)
423 if result is not None:
424 # query returns None if it is empty
425 queried_elements[key] = result
File /opt/conda/lib/python3.10/functools.py:889, in singledispatch.<locals>.wrapper(*args, **kw)
885 if not args:
886 raise TypeError(f'{funcname} requires at least '
887 '1 positional argument')
--> 889 return dispatch(args[0].__class__)(*args, **kw)
File /opt/conda/lib/python3.10/site-packages/spatialdata/_core/query/spatial_query.py:880, in _(points, polygon, target_coordinate_system, **kwargs)
877 polygon_gdf = _get_polygon_in_intrinsic_coordinates(points, target_coordinate_system, polygon)
879 points_gdf = points_dask_dataframe_to_geopandas(points, suppress_z_warning=True)
--> 880 joined = polygon_gdf.sjoin(points_gdf)
881 if len(joined) == 0:
882 return None
File /opt/conda/lib/python3.10/site-packages/geopandas/geodataframe.py:2391, in GeoDataFrame.sjoin(self, df, *args, **kwargs)
2307 def sjoin(self, df, *args, **kwargs):
2308 """Spatial join of two GeoDataFrames.
2309
2310 See the User Guide page :doc:`../../user_guide/mergingdata` for details.
(...)
2389 sjoin : equivalent top-level function
2390 """
-> 2391 return geopandas.sjoin(left_df=self, right_df=df, *args, **kwargs)
File /opt/conda/lib/python3.10/site-packages/geopandas/tools/sjoin.py:120, in sjoin(left_df, right_df, how, predicate, lsuffix, rsuffix, distance, on_attribute, **kwargs)
114 _basic_checks(left_df, right_df, how, lsuffix, rsuffix, on_attribute=on_attribute),
116 indices = _geom_predicate_query(
117 left_df, right_df, predicate, distance, on_attribute=on_attribute
118 )
--> 120 joined, _ = _frame_join(
121 left_df,
122 right_df,
123 indices,
124 None,
125 how,
126 lsuffix,
127 rsuffix,
128 predicate,
129 on_attribute=on_attribute,
130 )
132 return joined
File /opt/conda/lib/python3.10/site-packages/geopandas/tools/sjoin.py:494, in _frame_join(left_df, right_df, indices, distances, how, lsuffix, rsuffix, predicate, on_attribute)
492 new_index = pd.RangeIndex(len(l_idx))
493 left = left_df._reindex_with_indexers({0: (new_index, l_idx)})
--> 494 right = right_df._reindex_with_indexers({0: (new_index, r_idx)})
495 if PANDAS_GE_30:
496 kwargs = {}
File /opt/conda/lib/python3.10/site-packages/pandas/core/generic.py:5597, in NDFrame._reindex_with_indexers(self, reindexers, fill_value, copy, allow_dups)
5594 indexer = ensure_platform_int(indexer)
5596 # TODO: speed up on homogeneous DataFrame objects (see _reindex_multi)
-> 5597 new_data = new_data.reindex_indexer(
5598 index,
5599 indexer,
5600 axis=baxis,
5601 fill_value=fill_value,
5602 allow_dups=allow_dups,
5603 copy=copy,
5604 )
5605 # If we've made a copy once, no need to make another one
5606 copy = False
File /opt/conda/lib/python3.10/site-packages/pandas/core/internals/managers.py:670, in BaseBlockManager.reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups, copy, only_slice, use_na_proxy)
663 new_blocks = self._slice_take_blocks_ax0(
664 indexer,
665 fill_value=fill_value,
666 only_slice=only_slice,
667 use_na_proxy=use_na_proxy,
668 )
669 else:
--> 670 new_blocks = [
671 blk.take_nd(
672 indexer,
673 axis=1,
674 fill_value=(
675 fill_value if fill_value is not None else blk.fill_value
676 ),
677 )
678 for blk in self.blocks
679 ]
681 new_axes = list(self.axes)
682 new_axes[axis] = new_axis
File /opt/conda/lib/python3.10/site-packages/pandas/core/internals/managers.py:671, in <listcomp>(.0)
663 new_blocks = self._slice_take_blocks_ax0(
664 indexer,
665 fill_value=fill_value,
666 only_slice=only_slice,
667 use_na_proxy=use_na_proxy,
668 )
669 else:
670 new_blocks = [
--> 671 blk.take_nd(
672 indexer,
673 axis=1,
674 fill_value=(
675 fill_value if fill_value is not None else blk.fill_value
676 ),
677 )
678 for blk in self.blocks
679 ]
681 new_axes = list(self.axes)
682 new_axes[axis] = new_axis
File /opt/conda/lib/python3.10/site-packages/pandas/core/internals/blocks.py:1061, in Block.take_nd(self, indexer, axis, new_mgr_locs, fill_value)
1058 allow_fill = True
1060 # Note: algos.take_nd has upcast logic similar to coerce_to_target_dtype
-> 1061 new_values = algos.take_nd(
1062 values, indexer, axis=axis, allow_fill=allow_fill, fill_value=fill_value
1063 )
1065 # Called from three places in managers, all of which satisfy
1066 # these assertions
1067 if isinstance(self, ExtensionBlock):
1068 # NB: in this case, the 'axis' kwarg will be ignored in the
1069 # algos.take_nd call above.
File /opt/conda/lib/python3.10/site-packages/pandas/core/array_algos/take.py:115, in take_nd(arr, indexer, axis, fill_value, allow_fill)
110 arr = cast("NDArrayBackedExtensionArray", arr)
111 return arr.take(
112 indexer, fill_value=fill_value, allow_fill=allow_fill, axis=axis
113 )
--> 115 return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
117 arr = np.asarray(arr)
118 return _take_nd_ndarray(arr, indexer, axis, fill_value, allow_fill)
File /opt/conda/lib/python3.10/site-packages/pandas/core/arrays/arrow/array.py:1190, in ArrowExtensionArray.take(self, indices, allow_fill, fill_value)
1186 return result
1187 # return type(self)(pc.fill_null(result, pa.scalar(fill_value)))
1188 else:
1189 # Nothing to fill
-> 1190 return type(self)(self._pa_array.take(indices))
1191 else: # allow_fill=False
1192 # TODO(ARROW-9432): Treat negative indices as indices from the right.
1193 if (indices_array < 0).any():
1194 # Don't modify in-place
File /opt/conda/lib/python3.10/site-packages/pyarrow/table.pxi:1043, in pyarrow.lib.ChunkedArray.take()
File /opt/conda/lib/python3.10/site-packages/pyarrow/compute.py:486, in take(data, indices, boundscheck, memory_pool)
446 """
447 Select values (or records) from array- or table-like data given integer
448 selection indices.
(...)
483 ]
484 """
485 options = TakeOptions(boundscheck=boundscheck)
--> 486 return call_function('take', [data, indices], options, memory_pool)
File /opt/conda/lib/python3.10/site-packages/pyarrow/_compute.pyx:590, in pyarrow._compute.call_function()
File /opt/conda/lib/python3.10/site-packages/pyarrow/_compute.pyx:385, in pyarrow._compute.Function.call()
File /opt/conda/lib/python3.10/site-packages/pyarrow/error.pxi:154, in pyarrow.lib.pyarrow_internal_check_status()
File /opt/conda/lib/python3.10/site-packages/pyarrow/error.pxi:91, in pyarrow.lib.check_status()
ArrowInvalid: offset overflow while concatenating arrays
To Reproduce
sdata_crop = sd.polygon_query(
sdata_biopsy,
polygon=polygon,
target_coordinate_system='global',
)
Screenshots
Enviroment
-----
anndata 0.11.3
scanpy 1.10.4
-----
PIL 10.0.0
anyio NA
arrow 1.3.0
asciitree NA
asttokens NA
attr 22.2.0
awkward 2.7.2
awkward_cpp NA
babel 2.12.1
backcall 0.2.0
bottleneck 1.3.7
brotli NA
certifi 2024.12.14
cffi 1.15.1
charset_normalizer 3.1.0
cloudpickle 3.1.1
colorama 0.4.6
comm 0.1.3
cycler 0.10.0
cython_runtime NA
cytoolz 0.12.0
dask 2024.8.0
dask_expr 1.1.10
dask_image NA
datashader 0.16.3
datatree 0.0.14
dateutil 2.8.2
debugpy 1.6.7
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.6
docrep 0.3.2
exceptiongroup 1.2.2
executing 1.2.0
fastjsonschema NA
fqdn NA
fsspec 2023.4.0
geopandas 1.0.1
gmpy2 2.1.2
google NA
h5py 3.12.1
idna 3.4
igraph 0.11.2
imagecodecs 2023.8.12
imageio 2.28.0
ipykernel 6.22.0
ipython_genutils 0.2.0
ipywidgets 8.0.6
isal 1.7.1
isoduration NA
jedi 0.18.2
jinja2 3.1.2
joblib 1.4.2
json5 NA
jsonpointer 2.0
jsonschema 4.17.3
jupyter_events 0.6.3
jupyter_server 2.5.0
jupyterlab_server 2.22.1
kiwisolver 1.4.4
lazy_loader NA
legacy_api_wrap NA
leidenalg 0.10.2
llvmlite 0.43.0
lz4 4.3.2
markupsafe 2.1.2
matplotlib 3.7.1
matplotlib_inline 0.1.6
matplotlib_scalebar 0.9.0
mpl_toolkits NA
mpmath 1.3.0
msgpack 1.0.5
multipledispatch 0.6.0
multiscale_spatial_image 1.0.0
natsort 8.4.0
nbformat 5.8.0
networkx 3.1
numba 0.60.0
numcodecs 0.13.1
numexpr 2.8.4
numpy 1.26.4
ome_zarr NA
opt_einsum 3.4.0
packaging 23.1
pandas 2.1.0
param 2.2.0
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pkg_resources NA
platformdirs 3.5.0
plotly 5.24.1
pooch v1.7.0
prometheus_client NA
prompt_toolkit 3.0.38
psutil 5.9.5
ptyprocess 0.7.0
pure_eval 0.2.2
pvectorc NA
pyarrow 14.0.2
pycparser 2.21
pyct 0.5.0
pydev_ipython NA
pydevconsole NA
pydevd 2.9.5
pydevd_file_utils NA
pydevd_plugins NA
pydevd_tracing NA
pygments 2.15.1
pyparsing 3.0.9
pyproj 3.6.1
pyrsistent NA
pythonjsonlogger NA
pytz 2023.3
pywt 1.4.1
requests 2.29.0
rfc3339_validator 0.1.4
rfc3986_validator 0.1.1
rich NA
ruamel NA
scipy 1.11.4
seaborn 0.13.2
send2trash NA
session_info 1.0.0
setuptools 67.7.2
setuptools_scm NA
shapely 2.0.2
simplejson 3.20.1
six 1.16.0
skimage 0.20.0
sklearn 1.2.2
sniffio 1.3.0
socks 1.7.1
sparse 0.15.5
spatial_image 1.0.0
spatialdata 0.2.3
squidpy 1.6.2
stack_data 0.6.2
statsmodels 0.13.5
sympy 1.11.1
tblib 1.7.0
texttable 1.7.0
threadpoolctl 3.1.0
tifffile 2023.4.12
tlz 0.12.0
tomli 2.0.1
toolz 0.12.0
torch 2.0.0
tornado 6.3
tqdm 4.65.0
traitlets 5.9.0
typing_extensions NA
uri_template NA
urllib3 1.26.15
validators 0.34.0
wcwidth 0.2.6
webcolors NA
websocket 1.5.1
xarray 2023.12.0
xarray_dataclasses 1.7.0
xarray_schema 0.0.3
xrspatial 0.4.0
yaml 6.0.2
zarr 2.18.2
zipp NA
zmq 25.0.2
zoneinfo NA
-----
IPython 8.13.1
jupyter_client 8.2.0
jupyter_core 5.3.0
jupyterlab 3.6.3
notebook 6.5.4
-----
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:08:06) [GCC 11.3.0]
Linux-3.10.0-1160.81.1.el7.x86_64-x86_64-with-glibc2.35
-----
Session information updated at 2025-08-19 02:44
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at spatialdata/_core/query/spatial_query.py, especially _dict_query_dispatcher and the points polygon_query path around lines 831–880. Reproduce the supplied polygon_query call with the Xenium data and trace the GeoPandas join into pandas and Arrow selection shown in the traceback; done means the query completes without ArrowInvalid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100