Megvii-BaseDetection / Megvii-BaseDetection/YOLOX
Multi Gpu train
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
hello.
Please understand that I am using a translator as I am not good at English.
I have a problem in learning using multi gpus.
Learning using multi gpu is not possible on some GPUs.
The models with the current problem are quadro RTX6000 and RTX A6000.
However, in quadro rtx8000 gpu, learning using multi gpu is possible.
The only difference between the two computers is whether the cpu is intel(quadro RTX8000) or AMD(quadro RTX6000, RTX A6000)
When learning using multi gpu, it stops with the following console message and my computer starts to freeze.
yolox.core.launch:_distributed_worker:119 - Rank 0 initialization finished.
yolox.core.launch:_distributed_worker:119 - Rank 1 initialization finished.
I tested it on version and they all showed the same symptoms.
pytorch 1.7.1, 1.8.0, 1.10.0
cuda 10.2, 11.0, 11.1
Please answer these questions.
thank you.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at yolox.core.launch:_distributed_worker around line 119 and review the distributed initialization path. Reproduce multi-GPU training with the RTX6000 or RTX A6000 setup, comparing the reported PyTorch and CUDA versions with the working RTX8000 setup. Done means identifying a reproducible cause and documenting or fixing the initialization freeze.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100