scverse / scverse/spatialdata-io
PyArrow OSError: Unexpected end of stream
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 103
- Forks
- 65
- Avg merge
- 1h 8m
- Merged PRs (30d)
- 3
Description
Since the release of pyarrow 22.0.0 the CI started failing, with errors similar to the one below. Please note that the error doesn't seem to be spatialdata-related. Full CI run here.
tests/test_xenium.py:213:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/spatialdata_io/_utils.py:59: in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:192: in xenium
return_values = _get_tables_and_circles(path, cells_as_circles, specs, gex_only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:570: in _get_tables_and_circles
metadata = pd.read_parquet(path / XeniumKeys.CELL_METADATA_FILE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:669: in read_parquet
return impl.read(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:265: in read
pa_table = self.api.parquet.read_table(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1899: in read_table
return dataset.read(columns=columns, use_threads=use_threads,
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1538: in read
table = self._dataset.to_table(
pyarrow/_dataset.pyx:589: in pyarrow._dataset.Dataset.to_table
???
pyarrow/_dataset.pyx:3969: in pyarrow._dataset.Scanner.to_table
???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OSError: Unexpected end of stream
pyarrow/error.pxi:92: OSError
The pyarrow release also broke some tests in spatialdata, but these have been already fixed here: https://github.com/scverse/spatialdata/pull/1003, and a release will be made soon.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure from tests/test_xenium.py:213 using pyarrow 22.0.0 and inspect the parquet read in src/spatialdata_io/readers/xenium.py, especially _get_tables_and_circles. Compare the CI run and the related spatialdata fix, then confirm the Xenium test and CI pass with the resolved dependency behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100