scverse / scverse/rapids-singlecell

[BUG] running the Leiden algorithm doesn't support oversubscription

Open
#171 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
394
Forks
52
Avg merge
22h 5m
Merged PRs (30d)
20

Description

Describe the bug
While running the leiden algorithm rsc.tl.leiden(adata, key_added="leiden_res0_25", resolution=0.25 , I got a "CUDA error encountered 101 cudaErrorInvalidDevice invalid device ordinal"

Just setting rmm.reinitialize managed_memory to False resolved the issue

Expected behavior
Just information for other people running into the same error

Environment details (please complete the following information):

  • Environment location: Conda running in WSL2
  • Linux Distro/Architecture: Ubuntu 22.04.4 LTS
  • GPU Model/Driver: [RTX 3070 and driver 31.0.15.5161]
  • CUDA: 12.4
  • Method of Rapids install: conda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the rsc.tl.leiden call and the reported CUDA error in the WSL2, Conda, CUDA 12.4 environment. Check how rmm.reinitialize manages managed_memory and whether the workaround should be documented or the oversubscription behavior fixed. Done should be defined as either a reproducible fix or clear user-facing guidance for this configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.