Megvii-BaseDetection / Megvii-BaseDetection/YOLOX

Multi Gpu train

Open
#955 7 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10.6k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

hello.

Please understand that I am using a translator as I am not good at English.

I have a problem in learning using multi gpus.
Learning using multi gpu is not possible on some GPUs.

The models with the current problem are quadro RTX6000 and RTX A6000.
However, in quadro rtx8000 gpu, learning using multi gpu is possible.
The only difference between the two computers is whether the cpu is intel(quadro RTX8000) or AMD(quadro RTX6000, RTX A6000)

When learning using multi gpu, it stops with the following console message and my computer starts to freeze.

yolox.core.launch:_distributed_worker:119 - Rank 0 initialization finished.
yolox.core.launch:_distributed_worker:119 - Rank 1 initialization finished.

I tested it on version and they all showed the same symptoms.
pytorch 1.7.1, 1.8.0, 1.10.0
cuda 10.2, 11.0, 11.1

Please answer these questions.
thank you.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at yolox.core.launch:_distributed_worker around line 119 and review the distributed initialization path. Reproduce multi-GPU training with the RTX6000 or RTX A6000 setup, comparing the reported PyTorch and CUDA versions with the working RTX8000 setup. Done means identifying a reproducible cause and documenting or fixing the initialization freeze.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.