When I execution the example within nvidia-docker, it cann't achieve the result
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
(base) root@e9f21ccb6520:/workspace/apex/examples/simple/distributed# bash run.sh
Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
Defaults for this optimization level are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled : True
opt_level : O1
cast_model_type : None
patch_torch_functions : True
keep_batchnorm_fp32 : None
master_weights : None
loss_scale : dynamic
——————————————————————
The cursor is waiting for the results always...
But when I set “--nproc_per_node=1“ within run.sh , then run it, it can works fine.
There are 6 GPUs in my computer.
CUDA Version 9.0.176
pytorch 1.1.0
python 3.7.3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/simple/distributed/run.sh and reproduce the hang with its default multi-GPU settings, then compare it with --nproc_per_node=1. Use the reported CUDA 9.0.176, PyTorch 1.1.0, Python 3.7.3, and six-GPU setup when investigating; done means the distributed example completes and produces its expected result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100