write_element(name, overwrite=True) can never overwrite an existing element, even purely in-memory tables/shapes

[!NOTE]
This whole message is AI-generated. The issue was automatically discovered and reported by an AI agent (Claude) during an autonomous bug hunt on the spatialdata code base. It has not been verified or triaged by a human yet; the needs: triage label is set so that a maintainer can confirm it. The reproduction script below was executed by the agent in an isolated environment (see Environment) and its output is pasted verbatim.

Summary

Overwriting a Dask-backed element in place is rightly refused, but the same refusal applies to a plain in-memory AnnData table or GeoDataFrame, where no backing files are involved (_backed_elements_contained_in_path already knows this). Users must follow the workarounds of discussion #520 (write under a new name and rename, or delete_element_from_disk + write_element with a data-loss window).

Severity (agent's assessment): medium (usability / misleading API) — the behaviour is intentional and tested, so this is filed as a missing feature

Where: src/spatialdata/_core/spatialdata.py::_validate_can_safely_write_to_path (raises for any element path inside the object's own store when saving_an_element=True) and the write_element docstring ('overwrite: If True, overwrite the element if it already exists')

Expected behaviour

In-place overwrite for elements without Dask backing inside the target path (write to a temporary sibling group, then atomic replace + re-consolidate); keep the refusal for Dask-backed elements. Until then, document in the overwrite docstring that overwriting inside the backing store is refused.

Reproduction

Save as repro.py and run uv run repro.py (the PEP 723 header pins spatialdata to the commit the bug was found on; replace the URL fragment with @main to test the current main branch).

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "spatialdata @ git+https://github.com/scverse/spatialdata.git@ccf1ea048d054b6624214bf618008a9f9ae223e0",
# ]
# ///
"""write_element(name, overwrite=True) refuses to overwrite even purely in-memory elements (tables, shapes)."""
import os
import shutil
import tempfile
import warnings
import numpy as np
import pandas as pd
import geopandas as gpd
from anndata import AnnData
from shapely.geometry import Point
from spatialdata import SpatialData, read_zarr
from spatialdata.models import ShapesModel, TableModel

warnings.simplefilter("ignore")
tmp = tempfile.mkdtemp()
store = os.path.join(tmp, "store.zarr")
shapes = ShapesModel.parse(gpd.GeoDataFrame({"geometry": [Point(0, 0)], "radius": [1.0]}))
obs = pd.DataFrame({"region": pd.Categorical(["shp"]), "instance_id": [0]})
table = TableModel.parse(AnnData(X=np.zeros((1, 1)), obs=obs), region="shp", region_key="region", instance_key="instance_id")
SpatialData(shapes={"shp": shapes}).write(store)
sdata = read_zarr(store)
sdata.tables["t"] = table
sdata.write_element("t")
print("first write_element('t'): OK")
bug = False
try:
    sdata.write_element("t", overwrite=True)  # the table is a plain in-memory AnnData
    print("write_element('t', overwrite=True): OK")
except Exception as e:  # noqa: BLE001
    print("write_element('t', overwrite=True):", type(e).__name__, str(e).splitlines()[0][:100])
    bug = True
shutil.rmtree(tmp)
print("VERDICT:", "BEHAVIOUR REPRODUCED (overwrite refused for an in-memory table)" if bug else "NOT REPRODUCED")

Observed output

first write_element('t'): OK
write_element('t', overwrite=True): ValueError Cannot overwrite. The target path of the write operation is in use. Please save the data to a differ
VERDICT: BEHAVIOUR REPRODUCED (overwrite refused for an in-memory table)

Possible fix direction (unverified)

See above.

Environment

uv run repro.py with the PEP 723 metadata in the script (fresh, isolated environment; spatialdata built from main @ ccf1ea0 (2026-08-28); Python 3.13, latest releases of the dependencies at run time: pandas 3.0, anndata 0.13, zarr 3.3, dask 2026.8, numpy 2.5, geopandas 1.1, shapely 2.1). macOS (arm64). Also reproduced in a second environment with pandas 2.3.3 / anndata 0.12.11 / numpy 2.4.4 / zarr 3.2.1.

Possibly related issues

#866

Automatically generated; discovered by an AI agent (Claude) and not yet reviewed by a human.

Run the supplied repro.py with uv run repro.py, then inspect src/spatialdata/_core/spatialdata.py, especially _validate_can_safely_write_to_path and write_element. Confirm the distinction between in-memory and Dask-backed elements; done means in-memory tables or shapes can be overwritten while Dask-backed elements remain protected, with the overwrite documentation reflecting the behavior.