deepmodeling / deepmodeling/unimol_tools

[Code scan] Use selected GPU count for DDP world size and spawning

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34
Forks
9
PR merge metrics
No merged PRs in 30d

Description

This issue is a result of a Codex global repository scan.

## Summary
The trainer derives WORLD_SIZE and mp.spawn nprocs from torch.cuda.device_count(), even when use_gpu selects a subset such as "0,1". CUDA_VISIBLE_DEVICES is also mutated after querying CUDA, making selected-device and spawned-rank counts diverge.

## Code references
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L71-L94
https://github.com/deepmodeling/unimol_tools/blob/4596596aa8f73eb462d5cc5a921d79966d0465da/unimol_tools/tasks/trainer.py#L185-L203

## Impact
On hosts with more GPUs than requested, DDP can spawn too many processes or set a WORLD_SIZE that does not match the selected visible devices. This can fail initialization or leave ranks bound to unintended GPUs.

## Suggested fix
Parse use_gpu before DDP setup and set WORLD_SIZE and nprocs to the selected GPU count. Alternatively require users to set CUDA_VISIBLE_DEVICES or launch with torchrun, and avoid mutating CUDA_VISIBLE_DEVICES after CUDA has already been queried.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in unimol_tools/tasks/trainer.py at lines 71-94 and 185-203, tracing how use_gpu, CUDA visibility, WORLD_SIZE, and mp.spawn nprocs are determined. Verify the selected GPU count is used consistently before DDP setup, then run the trainer's existing tests or a minimal DDP launch to confirm ranks match the requested devices.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.