Opening a Visium HD store costs 7 GB of RAM because shapes and tables are loaded eagerly (lazy shapes/tables missing)

[!NOTE]
This whole message is AI-generated. The issue was automatically discovered and reported by an AI agent (Claude) during an autonomous bug hunt on the spatialdata code base. It has not been verified or triaged by a human yet; the needs: triage label is set so that a maintainer can confirm it. The reproduction script below was executed by the agent in an isolated environment (see Environment) and its output is pasted verbatim.

Summary

Measured on visium_hd_3.0.0_io from spatialdata-sandbox (5.48 M 2-µm bins as polygons + a 5.48 M × 19 k table), peak RSS of a fresh process:

read_zarr (everything)                    4.0 s   7.31 GB
read_zarr selection=("images","shapes")   1.7 s   4.24 GB   (5.5 M polygons in memory)
read_zarr selection=("tables",)           2.6 s   4.17 GB   (128 M non-zeros)

The 22k × 24k multiscale image costs nothing to open (lazy). selection only works per element type, so the 2-µm layer cannot be skipped while keeping the 8-µm one.

Severity (agent's assessment): high for real data — read_zarr() (hence SpatialData.read, peek, napari/plot loading) materialises every GeoDataFrame and AnnData even when only an image is needed; 8 GB machines cannot open the store

Where: src/spatialdata/_io/io_zarr.py::read_zarr → _read_shapes (geopandas.read_parquet) and _read_table (anndata.read_zarr)

Expected behaviour

Opening a store should be cheap; heavy elements should be loadable lazily or selectively.

Reproduction

Requires the visium_hd_3.0.0_io dataset from https://github.com/giovp/spatialdata-sandbox; pass its data.zarr path as the first argument.

Save as repro.py and run uv run repro.py (the PEP 723 header pins spatialdata to the commit the bug was found on; replace the URL fragment with @main to test the current main branch).

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "spatialdata @ git+https://github.com/scverse/spatialdata.git@ccf1ea048d054b6624214bf618008a9f9ae223e0",
# ]
# ///
"""read_zarr loads all shapes and tables eagerly: opening a Visium HD store costs several GB.

Needs a real dataset: `visium_hd_3.0.0_io` from https://github.com/giovp/spatialdata-sandbox
(5.48 M 2-um bins as polygons + 5.48 M x 19 k table). Usage: uv run repro.py /path/to/visium_hd_3.0.0_io/data.zarr
Each probe runs in a fresh subprocess and reports wall time and peak RSS.
"""
import json
import subprocess
import sys
import textwrap

path = sys.argv[1] if len(sys.argv) > 1 else "/Users/macbook/embl/projects/basel/spatialdata-sandbox/visium_hd_3.0.0_io/data.zarr"
TEMPLATE = textwrap.dedent("""
    import json, resource, time, warnings
    warnings.simplefilter("ignore")
    from spatialdata import read_zarr
    t0 = time.time()
    sdata = read_zarr({path!r}, selection={selection})
    n = len(list(sdata.gen_elements()))
    print(json.dumps(dict(seconds=round(time.time() - t0, 1), peak_rss_gb=round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1e9, 2), n_elements=n)))
""")
results = {}
for label, selection in [("everything", "None"), ("images only", "('images',)"), ("images + shapes", "('images', 'shapes')"), ("tables only", "('tables',)")]:
    res = subprocess.run([sys.executable, "-c", TEMPLATE.format(path=path, selection=selection)], capture_output=True, text=True)
    lines = [l for l in res.stdout.splitlines() if l.startswith("{")]
    results[label] = json.loads(lines[-1]) if lines else {"error": res.stderr.strip().splitlines()[-1][:200] if res.stderr.strip() else "no output"}
    print(f"read_zarr selection={label:16s} -> {results[label]}", flush=True)
bug = results.get("everything", {}).get("peak_rss_gb", 0) > 4
print("VERDICT:", "BUG REPRODUCED (opening the store needs several GB)" if bug else "NOT REPRODUCED / dataset not available")

Observed output

read_zarr selection=everything       -> {'seconds': 4.2, 'peak_rss_gb': 7.53, 'n_elements': 10}
read_zarr selection=images only      -> {'seconds': 0.0, 'peak_rss_gb': 0.25, 'n_elements': 4}
read_zarr selection=images + shapes  -> {'seconds': 1.5, 'peak_rss_gb': 4.23, 'n_elements': 7}
read_zarr selection=tables only      -> {'seconds': 2.6, 'peak_rss_gb': 4.15, 'n_elements': 3}
VERDICT: BUG REPRODUCED (opening the store needs several GB)

Possible fix direction (unverified)

Tables: opt-in read_zarr(..., lazy_tables=True) using anndata.experimental.read_lazy / read_elem_lazy. Shapes: a lazy or on-demand GeoParquet mode (see #359), or at least selection by element name. Document the memory cost in the Visium HD tutorial.

Environment

uv run repro.py with the PEP 723 metadata in the script (fresh, isolated environment; spatialdata built from main @ ccf1ea0 (2026-08-28); Python 3.13, latest releases of the dependencies at run time: pandas 3.0, anndata 0.13, zarr 3.3, dask 2026.8, numpy 2.5, geopandas 1.1, shapely 2.1). macOS (arm64).

Possibly related issues

#359, #293

Automatically generated; discovered by an AI agent (Claude) and not yet reviewed by a human.

Run repro.py with the visium_hd_3.0.0_io dataset to confirm the memory measurements. Then read src/spatialdata/_io/io_zarr.py, especially read_zarr, _read_shapes, and _read_table, along with related issues #359 and #293. Done means heavy shapes and tables can be loaded lazily or selectively without materializing them when opening an image-focused store.