read_h5ad backed mode still runs out of memory on large datasets

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of scanpy.
• (optional) I have confirmed this bug exists on the master branch of scanpy.

I am not 100% this is a bug, so please correct me if I'm doing something wrong...

I am using scanpy with a very large scRNAseq dataset (SEA-AD, the sparse h5ad is ~35GB)
When I use the read_h5ad file, even when running in backed mode, I get an out of memory exception.

I expect it to be due to the fact that parts of the dataset are still read to memory, even in backed mode.
As you can see in the stack trace below, eventually anndata's read_sparse() function is called (in _io/specs/methods.py
But this method has the following implementation in the latest version:

def read_sparse(elem):
    return SparseDataset(elem).to_memory()

Thus, loading (part of) the dataset to memory.

Like I said, I am not sure whether this is a bug, or supposed to happen. But to me it seems odd that backed mode still loads large portions of the dataset to memory.

Workaround

For my own project, I got around this issue, by removing the call to .to_memory() within the source of anndata. I am not sure whether this breaks any other functionality, but I can use the dataset the way I need it right now.

Minimal code sample (that we can copy&paste without having any data)

(this cannot be run without any data, because the problem is that it fails with a big dataset. I included an aws download command to the big dataset that is causing the crash on my 24GB memory machine)

import scanpy

# Download command
# aws s3 cp --no-sign-request s3://sea-ad-single-cell-profiling/MTG/RNAseq/SEAAD_MTG_RNAseq_final-nuclei.2022-08-18.h5ad ./final.h5ad

PATH = './final.h5py'
adata = scanpy.read_h5ad(PATH, backed=True)

(the stack trace below was redacted a bit to hide my private information. [python-path] replaces the path to pythons directory)

Traceback (most recent call last):
  File "scanpy_test.py", line 9, in <module>
    adata = sc.read_h5ad(PATH, backed=True)
  File "[python-path]\site-packages\anndata\_io\h5ad.py", line 202, in read_h5ad
    return read_h5ad_backed(filename, mode)
  File "[python-path]\site-packages\anndata\_io\h5ad.py", line 146, in read_h5ad_backed
    d.update({k: read_elem(f[k]) for k in attributes if k in f})
  File "[python-path]\site-packages\anndata\_io\h5ad.py", line 146, in <dictcomp>
    d.update({k: read_elem(f[k]) for k in attributes if k in f})
  File "[python-path]\site-packages\anndata\_io\specs\registry.py", line 183, in read_elem
    return _REGISTRY.get_reader(type(elem), get_spec(elem), frozenset(modifiers))(elem)
  File "[python-path\site-packages\anndata\_io\specs\methods.py", line 92, in read_basic
    return {k: read_elem(v) for k, v in elem.items()}
  File "[python-path]\site-packages\anndata\_io\specs\methods.py", line 92, in <dictcomp>
    return {k: read_elem(v) for k, v in elem.items()}
  File "[python-path]\site-packages\anndata\_io\specs\registry.py", line 183, in read_elem
    return _REGISTRY.get_reader(type(elem), get_spec(elem), frozenset(modifiers))(elem)
  File "[python-path]\site-packages\anndata\_io\specs\methods.py", line 475, in read_sparse
    return SparseDataset(elem).to_memory()
  File "[python-path]\site-packages\anndata\_core\sparse_dataset.py", line 380, in to_memory
    mtx.indices = self.group["indices"][...]
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "[python-path]\site-packages\h5py\_hl\dataset.py", line 741, in __getitem__
    return self._fast_reader.read(args)
  File "h5py\_selector.pyx", line 362, in h5py._selector.Reader.read
  File "h5py\_selector.pyx", line 336, in h5py._selector.Reader.make_array
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 56.8 GiB for an array with shape (7621750342,) and data type int64

Process finished with exit code 1

Versions

Start with scanpy.read_h5ad(..., backed=True), then inspect anndata/_io/specs/methods.py at read_sparse and anndata/_core/sparse_dataset.py at to_memory. Reproduce with the large SEA-AD h5ad example and verify that backed loading no longer attempts the reported 56.8 GiB allocation while preserving expected dataset access.