Lightning-AI / Lightning-AI/pytorch-lightning
Support BatchSizeFinder in DDP
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Based on the docs, BatchSizeFinder does not currently support DDP:
>Batch size finder is not yet supported for DDP or any of its variations, it is coming soon.
https://lightning.ai/docs/pytorch/stable/advanced/training_tricks.html
It would be great to have a way to automatically find the correct batch size when using submitting DDP jobs to a queue that has multiple instance types. We would like to automatically detect the correct batch size to use when doing single instance, multi-gpu training.
### Pitch
Support BatchSizeFinder when using DDP strategy.
### Alternatives
Alternatively, provide some way to run BatchSizeFinder on a single GPU prior to DDP being used.
### Additional context
Related Issues and PRs:
- https://github.com/Lightning-AI/pytorch-lightning/issues/10560
- https://github.com/Lightning-AI/pytorch-lightning/pull/10754
- https://github.com/Lightning-AI/pytorch-lightning/pull/11089
- https://github.com/Lightning-AI/pytorch-lightning/pull/12179
- https://github.com/Lightning-AI/pytorch-lightning/pull/15305
cc @lantiga @borda @justusschock
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the BatchSizeFinder and DDP strategy entry points, then review the linked documentation and related issues and pull requests for prior design decisions. Done means BatchSizeFinder can automatically determine a usable batch size for DDP jobs and the documentation no longer says DDP is unsupported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100