scverse / scverse/spatialdata

write_element and delete_element_from_disk are very slow when SpatialData object contains large number of elements

Open
#938 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs: triage priority: low
Dominant language
Python
Stars
394
Forks
95
Avg merge
4d 3h
Merged PRs (30d)
7

Description

I work on a dataset with 1000 images and create and delete a lot of labels for these images. Writing and deleting label objects via write_element / delete_element_from_disk can take up to 60s per element when >10k elements are in the SpatialData object. The slowdown mostly happens in elements_paths_on_disk in spatialdata._core.spatialdata. Following change helped for me fix the issue, saving ~1000 labels in 40s:

def elements_paths_on_disk(self) -> list[str]:
    """
    Get the paths of the elements saved in the Zarr store.

    Returns
    -------
    A list of paths of the elements saved in the Zarr store.
    """
    if self.path is None:
        raise ValueError("The SpatialData object is not backed by a Zarr store.")
    store = parse_url(self.path, mode="r").store
    elements_in_zarr = []

    groups_stored = store.listdir()
    for group in groups_stored:
        if group in ["images", "labels", "points", "shapes"]:
            group_elems = [os.path.join(group, elem) for elem in store.listdir(group)]
            elements_in_zarr.extend(group_elems)
    return elements_in_zarr

In delete_element_from_disk calling write_consolidate_metadata takes a long time( ~1min). When expecting to delete a lot of images other users should call sdata.write() with consolidate_metadata=False, or rewrite delete_elements_from_disk such that when given a list of elements to delete write_consolidated_metadata is called only once at the end of the list.

Code to reproduce problem:

from spatialdata.datasets import blobs
import numpy as np
import time
import spatialdata as sd

sdata = blobs()
sdata.write('test', consolidate_metadata=True)

test = np.empty((1,1), dtype=np.uint8)
for i in range(1500):
    sdata[f'test{i}'] = sd.models.Labels2DModel().parse(test, dims=('y','x'))
    start = time.time()
    sdata.write_element(f'test{i}')
    print(f'Wrote test{i} in ', time.time()-start)
for i in range(1500):
    start = time.time()
    sdata.delete_element_from_disk(f'test{i}')
    print(f'Deleted test{i} in ', time.time()-start)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in spatialdata/_core/spatialdata.py, focusing on elements_paths_on_disk, write_element, and delete_element_from_disk. Run the supplied 1,500-element reproduction to measure writing and deletion times, then verify the chosen change preserves on-disk element handling while avoiding repeated expensive work. Confirm the result with the same reproduction and metadata-consolidation scenarios described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.