Opening remote data store is inconsistent
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
This works:
import zarr
import spatialdata as sd
root = zarr.open(
"s3://bucket_name/spatial-sandbox/visium_associated_xenium_io.zarr",
storage_options={
"endpoint_url": "https://objectstor.vib.be",
},
)
sdata = sd.SpatialData.read(root)
# sdata has content
This does not work:
import zarr
import spatialdata as sd
import s3fs
s3_out = s3fs.S3FileSystem(endpoint_url="https://objectstor.vib.be")
fsmap = s3fs.S3Map(
root="s3://bucket_name/spatial-sandbox/visium_associated_xenium_io.zarr", s3=s3_out, check=False
)
zarr_store = zarr.group(store=fsmap)
sdata = sd.SpatialData.read(zarr_store)
# sdata has no content
Again a parsing issue probably. Refactoring to https://github.com/fsspec/universal_pathlib will maybe fix a lot of these issues and allow more flexible configurations like cached stores.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the two paths using zarr.open with storage_options and s3fs.S3Map passed to zarr.group, then compare what SpatialData.read receives. Investigate the suggested universal_pathlib refactor and define done as consistent loading of the remote store, including the content currently missing from the S3Map path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100