scverse / scverse/spatialdata

Allow reloading of particular elements, and add `reload: bool` parameter to write functions

Open
#521 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

I/O 💿
Dominant language
Python
Stars
394
Forks
95
Avg merge
4d 3h
Merged PRs (30d)
7

Description

With the new incremental IO PR we can write single elements or metadata into a Zarr store. However, reading is still a monolithic operation that reads all at once.

We should

  • refactor the code for reading into individual components to read single elements and metadata of single elements
  • add new APIs to reload the data, metadata and in particular transformations of a single element in a SpatialData object.
  • consider adding a parameter return_stored (as discussed here https://docs.dask.org/en/latest/generated/dask.array.store.html#dask.array.store), or maybe reload, to allow the user to call write_element() and write() and return an object that is already reloaded with Dask. write(reload=True) is equivalent to calling sdata = read_zarr(sdata.path)
  • write_element(reload=True) would introduce a new convenient functionality that has for instance been mentioned by @ArneDefauw here https://github.com/scverse/spatialdata/pull/501#issuecomment-2018315221, and could be used to simplify in particular the workaround 1 for overwriting elements (test_incremental_io_on_disk()).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing write_element(), write(), and read_zarr() entry points, then inspect test_incremental_io_on_disk() and the linked incremental IO work. Clarify the component-level read and reload API before implementation. Done means single elements and metadata can be reloaded and the proposed write reload behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.