deepmodeling / deepmodeling/unimol_tools
[Code scan] Use selected GPU count for DDP world size and spawning
- Dominant language
- Python
- Stars
- 34
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
This issue is a result of a Codex global repository scan.
## Summary
The trainer derives WORLD_SIZE and mp.spawn nprocs from torch.cuda.device_count(), even when use_gpu selects a subset such as "0,1". CUDA_VISIBLE_DEVICES is also mutated after querying CUDA, making selected-device and spawned-rank counts diverge.
## Code references
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L71-L94
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L185-L203
## Impact
On hosts with more GPUs than requested, DDP can spawn too many processes or set a WORLD_SIZE that does not match the selected visible devices. This can fail initialization or leave ranks bound to unintended GPUs.
## Suggested fix
Parse use_gpu before DDP setup and set WORLD_SIZE and nprocs to the selected GPU count. Alternatively require users to set CUDA_VISIBLE_DEVICES or launch with torchrun, and avoid mutating CUDA_VISIBLE_DEVICES after CUDA has already been queried.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in unimol_tools/tasks/trainer.py at lines 71-94 and 185-203, tracing how use_gpu, CUDA visibility, WORLD_SIZE, and mp.spawn nprocs are determined. Verify the selected GPU count is used consistently before DDP setup, then run the trainer's existing tests or a minimal DDP launch to confirm ranks match the requested devices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100