scverse / scverse/rapids-singlecell
[FEA] Multi-GPU UMAP calculations to bypass Float64 nearest neighbor issue
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 52
- Avg merge
- 22h 5m
- Merged PRs (30d)
- 20
Description
Is your feature request related to a problem? Please describe.
Would it be possible to enable multi-GPU streamed UMAP calculations?
Some of the existing algorithms for CSR-based cuML don't work with float 64 (only float32) and as single cell datasets get bigger, the nearest neighbor graphs surpass float32 limitations and don't always fit in smaller scale GPUs.
Describe the solution you'd like
Nearest neighbor graphs would be streaming from CSR host into multiple GPUs, with regional estimations.
This would enable the nearest neighbor graphs to stay within the float32 limits, and avoid the recurrent float64 error in CuPy CSR matrices.
It would also allow GPU-accelerated UMAP calculations to be done on massive datasets via smaller GPU clusters, like L40S or A10Gs.
Is there a CPU based implementation
I don't believe there is a CPU-based implementation?
but there is an option available within Nvidia's existing tech stack
https://developer.nvidia.com/blog/even-faster-and-more-scalable-umap-on-the-gpu-with-rapids-cuml/
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing UMAP and CSR-based nearest-neighbor paths, then review the linked RAPIDS cuML design for streamed multi-GPU calculations. Done should include multi-GPU regional processing of large CSR datasets while avoiding the reported float64 CuPy CSR failure and float32 graph-size limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100