Megvii-BaseDetection / Megvii-BaseDetection/YOLOX
单机多卡训练，总是出错
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
2022-08-11 16:58:36.000 [INFO] [Driver] File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 274, in _get_group_size
2022-08-11 16:58:36.000 [INFO] [Driver] default_pg = _get_default_group()
2022-08-11 16:58:36.000 [INFO] [Driver] └ <function _get_default_group at 0x7f5aae921620>
2022-08-11 16:58:36.000 [INFO] [Driver] File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 358, in _get_default_group
2022-08-11 16:58:36.000 [INFO] [Driver] raise RuntimeError("Default process group has not been initialized, "
2022-08-11 16:58:36.000 [INFO] [Driver]
2022-08-11 16:58:36.000 [INFO] [Driver] RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The traceback is in torch/distributed/distributed_c10d.py, especially _get_group_size and _get_default_group. Start by reproducing the single-machine multi-GPU training command and trace which YOLOX entry point reaches these calls. Done means the training run initializes the process group before querying its size and completes without this RuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100