Categorical aggregate() (transcripts → cells count matrix) stores every (cell, gene) zero explicitly: memory ∝ n_cells × n_genes

[!NOTE]
This whole message is AI-generated. The issue was automatically discovered and reported by an AI agent (Claude) during an autonomous bug hunt on the spatialdata code base. It has not been verified or triaged by a human yet; the needs: triage label is set so that a maintainer can confirm it. The reproduction script below was executed by the agent in an isolated environment (see Environment) and its output is pasted verbatim.

Summary

For a categorical value_key, _aggregate_shapes groups with observed=False, which expands every observed region × every category, and the resulting zeros are written into the COO/CSR matrix (X.nnz counts them, they are never eliminated). On the Xenium xenium_rep1_io dataset (42.6 M transcripts, 167,780 cells, 541 features) the result has 90,087,861 stored entries of which only 12,020,857 are non-zero (13 %), peak RSS 16 GB, 44.6 s (measured after working around the Scale.__eq__ crash reported separately). The synthetic script below shows the same pattern at small scale.

Severity (agent's assessment): high for real data — a 5k-gene panel with 500k cells implies 2.5e9 stored entries (~30 GB) although the count matrix has a few 1e7 non-zeros

Where: src/spatialdata/_core/operations/aggregate.py::_aggregate_shapes (groupby([INDEX, vk], observed=False) followed by a COO construction that keeps the zeros)

Expected behaviour

Stored entries ≈ true non-zeros; memory proportional to the number of transcripts.

Reproduction

Save as repro.py and run uv run repro.py (the PEP 723 header pins spatialdata to the commit the bug was found on; replace the URL fragment with @main to test the current main branch).

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "spatialdata @ git+https://github.com/scverse/spatialdata.git@ccf1ea048d054b6624214bf618008a9f9ae223e0",
# ]
# ///
"""Categorical aggregate() materialises every (region, category) pair: the 'sparse' result stores mostly zeros."""
import time
import tracemalloc
import warnings
import numpy as np
import pandas as pd
import geopandas as gpd
from shapely.geometry import box
from spatialdata import aggregate
from spatialdata.models import PointsModel, ShapesModel

warnings.simplefilter("ignore")
rng = np.random.default_rng(0)
n_points, n_categories, side = 300_000, 500, 141  # ~20k cells
points = PointsModel.parse(
    pd.DataFrame({"x": rng.uniform(0, side, n_points), "y": rng.uniform(0, side, n_points),
                  "gene": pd.Categorical(rng.integers(0, n_categories, n_points).astype(str))}),
    feature_key="gene",
)
cells = ShapesModel.parse(gpd.GeoDataFrame({"geometry": [box(i, j, i + 1, j + 1) for i in range(side) for j in range(side)]}))
tracemalloc.start()
t0 = time.time()
out = aggregate(values=points, by=cells, value_key="gene", agg_func="count")
elapsed = time.time() - t0
_, peak = tracemalloc.get_traced_memory()
X = out["table"].X
true_nnz = int((X != 0).sum())
print(f"{n_points:,} points, {n_categories} categories, {len(cells):,} cells: {elapsed:.1f}s, peak traced memory {peak / 1e9:.2f} GB")
print(f"X.nnz (stored entries) = {X.nnz:,}   true non-zeros = {true_nnz:,}   ({100 * true_nnz / X.nnz:.1f}% of stored entries are non-zero)")
print("expected: stored entries ~= true non-zeros (memory proportional to the number of transcripts, not cells x genes)")
bug = X.nnz > 5 * true_nnz
print("VERDICT:", "BUG REPRODUCED" if bug else "NOT REPRODUCED")

Observed output

300,000 points, 500 categories, 19,881 cells: 2.1s, peak traced memory 0.60 GB
X.nnz (stored entries) = 9,940,500   true non-zeros = 295,520   (3.0% of stored entries are non-zero)
expected: stored entries ~= true non-zeros (memory proportional to the number of transcripts, not cells x genes)
VERDICT: BUG REPRODUCED

Possible fix direction (unverified)

Use observed=True in the groupby (the COO assembly already handles missing pairs and pd.Categorical(..., categories=...) keeps the full column set), and call X.eliminate_zeros() / sum_duplicates(). Related to the lazy-aggregation tracking issue #210.

Environment

uv run repro.py with the PEP 723 metadata in the script (fresh, isolated environment; spatialdata built from main @ ccf1ea0 (2026-08-28); Python 3.13, latest releases of the dependencies at run time: pandas 3.0, anndata 0.13, zarr 3.3, dask 2026.8, numpy 2.5, geopandas 1.1, shapely 2.1). macOS (arm64). Also reproduced in a second environment with pandas 2.3.3 / anndata 0.12.11 / numpy 2.4.4 / zarr 3.2.1.

Possibly related issues

#210, #744, #743

Automatically generated; discovered by an AI agent (Claude) and not yet reviewed by a human.

Start in src/spatialdata/_core/operations/aggregate.py at _aggregate_shapes, focusing on the groupby and COO construction described in the issue. Run the provided repro.py with uv run to compare stored entries with true non-zeros. Done means categorical aggregation stores approximately the true non-zero entries while preserving the full category column set and existing aggregate behavior.