Allow reloading of particular elements, and add `reload: bool` parameter to write functions
Open
Nobody has claimed this yet.
I/O 💿
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
With the new incremental IO PR we can write single elements or metadata into a Zarr store. However, reading is still a monolithic operation that reads all at once.
We should
- refactor the code for reading into individual components to read single elements and metadata of single elements
- add new APIs to reload the data, metadata and in particular transformations of a single element in a
SpatialDataobject. - consider adding a parameter
return_stored(as discussed here https://docs.dask.org/en/latest/generated/dask.array.store.html#dask.array.store), or maybereload, to allow the user to callwrite_element()andwrite()and return an object that is already reloaded with Dask.write(reload=True)is equivalent to callingsdata = read_zarr(sdata.path) -
write_element(reload=True)would introduce a new convenient functionality that has for instance been mentioned by @ArneDefauw here https://github.com/scverse/spatialdata/pull/501#issuecomment-2018315221, and could be used to simplify in particular the workaround 1 for overwriting elements (test_incremental_io_on_disk()).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing write_element(), write(), and read_zarr() entry points, then inspect test_incremental_io_on_disk() and the linked incremental IO work. Clarify the component-level read and reload API before implementation. Done means single elements and metadata can be reloaded and the proposed write reload behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100