scverse / scverse/spatialdata

Output spatially ordered `anndata` object

Open
#1,145 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
394
Forks
95
Avg merge
4d 3h
Merged PRs (30d)
7

Description

Is your feature request related to a problem? Please describe.
I am not 100% certain if this request is supported from the parquet side for shape/points (only going based on https://spatialdata.scverse.org/en/stable/api/SpatialData.html#spatialdata.SpatialData.write) but I'm following up on https://scverse.zulipchat.com/#narrow/channel/315824-spatial/topic/dataloaders.20for.20spatial.20omics/near/591781853

Describe the solution you'd like
I want the anndata object to be ordered by some form of spatial coherence - I can think of two versions:

  1. A pure-spatial approach that relies on some form of quad-tree to do the partitioning and the data would be written in some ordering (C? morton?) of the spatial coordinates of each 4-element leaf node of the tree.
  2. A user-defined neighborhood approach that outputs an approximate ordering so that neighbors are near one-another according to a user-defined neighborhood graph

Describe alternatives you've considered
For the second option, we could upstream this into anndata theoretically or factor out the "generate the linearized in-memory ordering of the indices" part to be reusable. This might actually make more sense TBH but it's not immediately clear to me if there will be a large benefit of the second over the first aside from the fact that the second would be more generalizable to other use-cases outside of spatial data, at the downside of requiring a (spatial) graph to be formed (which requires some form of computation?).

Additional context
The need here is produce data whose ordering is amenable to fast data loading via "batched fetching" i.e., ensuring as many data points as possible (for performance reasons) are contiguous on-disk.

I am not super familiar with spatial workflows so perhaps one makes more sense than the other in a normal pipeline, but I basically suspect we will end up implementing both TBH.

cc @ori-kron-wis @timtreis

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SpatialData.write and the parquet handling for shape/points mentioned in the issue. Review how anndata ordering is currently represented, then compare the proposed spatial quad-tree ordering with the user-defined neighborhood approach. Done should include a decided ordering strategy and a clear way to verify improved batched fetching locality.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.