A error occoured when using slurm
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
I used srun to train example/pytorch_imagenet_resnet50.py on 8 GUPs and got this error:
$ srun --mpi=pmi2 -p Platform -n1 --gres=gpu:8 --ntasks-per-node=8 horovodrun -np 8 -H localhost:8 python pytorch_imagenet_resnet50.py
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 8 slots
that were requested by the application:
python
Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
But it can work on 1 GPU with command
srun --mpi=pmi2 -p Platform -n1 --gres=gpu:1 --ntasks-per-node=1 horovodrun -np 1 -H localhost:1 python pytorch_imagenet_resnet50.py
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported srun/horovodrun commands and the example/pytorch_imagenet_resnet50.py training entry point, comparing the one-GPU and eight-GPU cases. Check the project's guidance for Slurm and distributed execution, and confirm whether the reported slot error is within Apex's scope. Done when the cause and supported multi-GPU invocation are established or the issue is redirected appropriately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100