Is distributed package slower than nn.DataParallel?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
First of all, I really appreciate to NVIDIA team which made the apex packages.
I have a question about the speed between apex distributed module and torch's DataParallel module.
If I can train the dataset with batch_size 256 using 4 gpus which model uses DataParallel module, I can also train the dataset with batch_size 64 at each gpu (4 gpus * 64 batch_size = 256) which model uses apex distributed package, right? (ex, python -m torch.distributed.launch --nproc_per_node=4 train.py)
But in my experiment, the training time was different. it was about four times of the model using DataParallel module.
I'm curious about this result... Is it right? or did I make a mistake?
Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report names train.py and the torch.distributed.launch command but provides no other files, tests, or reproducible configuration. Start by comparing the DataParallel and distributed launch setups using the supplied command; done requires a documented, reproducible explanation of the timing difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100