facebookresearch / facebookresearch/dora
No training when using 2 nodes and torchrun
Open
bug
- Dominant language
- Python
- Stars
- 314
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for adding the ability to use multi-nodes without slurm! When I run training on one machine using torchrun, everything works without problems. But when I run it on two machines, the training freezes. The machines connect to each other, the model is loaded, but the training process does not continue. Learning gets stuck somewhere. Now I'm trying to figure out how to solve this problem. My launch command:
```
torchrun --master-addr [ip] \
--master-port [port] \
--node_rank 0 \
--nnodes 2 \
--nproc-per-node 2 \
-m dora run [ARGS]
```
Contributor guide
Assessment
This issue has not been assessed yet.