Lightning-AI / Lightning-AI/pytorch-lightning
[RFC] "auto" precision support
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
Introducing the support for `precision="auto"`, which enables AMP whenever available and falls back to fp32 otherwise.
### Motivation
After introducing `accelerator="auto"` support, it is hard to determine the precision type to use.
Personally, I use AMP, whenever running on GPU and the GPU supports it. Otherwise I use fp32. Since precision has to be set at the same time, we specify the accelerator, with `accelerator="auto"` we cannot easily at that time determine on which accelerator it will be running (well we could, but that would be kind of duplicating logic we use for accelerator-detection internally) without a larger amount of boilerplate.
### Pitch
Have `precision="auto"` to switch between AMP and fp32 depending on the accelerator and whether the accelerator on hand does support AMP.
### Alternatives
Manually write out all the boilerplate.
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
cc @borda @tchaton @justusschock @awaelchli @carmocca @akihironitta @rohitgr7 @kaushikb11 @ananthsub
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing accelerator="auto" support and the code that handles AMP precision selection. Determine where precision is resolved relative to accelerator detection, then verify that precision="auto" selects AMP when supported and fp32 otherwise with suitable tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100