scverse / scverse/spatialdata-io

PyArrow OSError: Unexpected end of stream

Open
#334 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
103
Forks
65
Avg merge
1h 8m
Merged PRs (30d)
3

Description

Since the release of pyarrow 22.0.0 the CI started failing, with errors similar to the one below. Please note that the error doesn't seem to be spatialdata-related. Full CI run here.

tests/test_xenium.py:213: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/spatialdata_io/_utils.py:59: in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:192: in xenium
    return_values = _get_tables_and_circles(path, cells_as_circles, specs, gex_only)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:570: in _get_tables_and_circles
    metadata = pd.read_parquet(path / XeniumKeys.CELL_METADATA_FILE)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:669: in read_parquet
    return impl.read(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:265: in read
    pa_table = self.api.parquet.read_table(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1899: in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1538: in read
    table = self._dataset.to_table(
pyarrow/_dataset.pyx:589: in pyarrow._dataset.Dataset.to_table
    ???
pyarrow/_dataset.pyx:3969: in pyarrow._dataset.Scanner.to_table
    ???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   OSError: Unexpected end of stream

pyarrow/error.pxi:92: OSError

The pyarrow release also broke some tests in spatialdata, but these have been already fixed here: https://github.com/scverse/spatialdata/pull/1003, and a release will be made soon.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure from tests/test_xenium.py:213 using pyarrow 22.0.0 and inspect the parquet read in src/spatialdata_io/readers/xenium.py, especially _get_tables_and_circles. Compare the CI run and the related spatialdata fix, then confirm the Xenium test and CI pass with the resolved dependency behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.