Definition of parallel and distributed training on multiple GPUs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
Hello,
(Sorry if this is a wrong place to post this, but since I have not found any proper explanation in other packages it will be a very simple answer for anyone who knows what I am talking about.)
PyTorch DataParallel splits the batch size according to how many GPUs I have on a single machine. This does not reduce the training time but allows for larger batch size.
So my question is: Can I run distributed training on the same machine on multiple GPUs? I.e. every GPU gets the same batch size as the original but runs the model in parallel, so that I get multiple gradient updates on a single step, i.e. faster training?
If this is feasible, does apex support it?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the PyTorch DataParallel behavior described in the issue and checking Apex's distributed-training support. Determine whether the requested same-machine, multiple-GPU training mode is supported, and identify the relevant Apex entry points or documentation. Done means providing a definitive feasibility and support answer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100