Output spatially ordered `anndata` object
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
Is your feature request related to a problem? Please describe.
I am not 100% certain if this request is supported from the parquet side for shape/points (only going based on https://spatialdata.scverse.org/en/stable/api/SpatialData.html#spatialdata.SpatialData.write) but I'm following up on https://scverse.zulipchat.com/#narrow/channel/315824-spatial/topic/dataloaders.20for.20spatial.20omics/near/591781853
Describe the solution you'd like
I want the anndata object to be ordered by some form of spatial coherence - I can think of two versions:
- A pure-spatial approach that relies on some form of quad-tree to do the partitioning and the data would be written in some ordering (C? morton?) of the spatial coordinates of each 4-element leaf node of the tree.
- A user-defined neighborhood approach that outputs an approximate ordering so that neighbors are near one-another according to a user-defined neighborhood graph
Describe alternatives you've considered
For the second option, we could upstream this into anndata theoretically or factor out the "generate the linearized in-memory ordering of the indices" part to be reusable. This might actually make more sense TBH but it's not immediately clear to me if there will be a large benefit of the second over the first aside from the fact that the second would be more generalizable to other use-cases outside of spatial data, at the downside of requiring a (spatial) graph to be formed (which requires some form of computation?).
Additional context
The need here is produce data whose ordering is amenable to fast data loading via "batched fetching" i.e., ensuring as many data points as possible (for performance reasons) are contiguous on-disk.
I am not super familiar with spatial workflows so perhaps one makes more sense than the other in a normal pipeline, but I basically suspect we will end up implementing both TBH.
cc @ori-kron-wis @timtreis
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with SpatialData.write and the parquet handling for shape/points mentioned in the issue. Review how anndata ordering is currently represented, then compare the proposed spatial quad-tree ordering with the user-defined neighborhood approach. Done should include a decided ordering strategy and a clear way to verify improved batched fetching locality.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100