Lightning-AI / Lightning-AI/pytorch-lightning
Support batch size scaling with dataloaders passed directly to `fit()`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
*This was first suggested/reported in #4000 and addressed in #4006 but then reverted in #4040 due to some GPU testing failure.*
Currently, batch size scaling cannot be used with dataloaders passed directly to `trainer.fit()` as exception is raised at:
https://github.com/PyTorchLightning/pytorch-lightning/blob/fe34bf2a653ebd50e6a3a00be829e3611f820c3c/pytorch_lightning/tuner/batch_size_scaling.py#L56-L58
The reason why this limitation was set is explained in #4006.
> This limitation was set in place, to secure that the user had a field self.batch_size that we could alter.
To remove this limitation we instead replaces the dataloader with a newly instantiated dataloader with altered batch size.
### Motivation
### Pitch
### Alternatives
### Additional context
- The limitation has been discovered in the current tuner revamp PR: https://github.com/PyTorchLightning/pytorch-lightning/pull/11089#discussion_r782849342.
- TODO after #11089 lands.
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
cc @borda @justusschock @awaelchli @ninginthecloud @rohitgr7 @otaj @akihironitta
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pytorch_lightning/tuner/batch_size_scaling.py at the exception around lines 56-58, then review the behavior discussed in issues #4000, #4006, and #4040 and the tuner revamp context in #11089. The work is done when batch size scaling accepts dataloaders passed directly to trainer.fit() and uses a newly instantiated dataloader with the altered batch size without the current exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100