Lightning-AI / Lightning-AI/pytorch-lightning
Support AMP with TPUs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Lightning currently supports `accelerator="tpu", precision="bf16-mixed"`, but so far, this just sets the `XLA_USE_BF16` environment variable:
- Trainer: https://github.com/Lightning-AI/lightning/blob/master/src/lightning/pytorch/plugins/precision/xlabf16.py
- Fabric: https://github.com/Lightning-AI/lightning/blob/master/src/lightning/fabric/plugins/precision/xlabf16.py
> Side note: why does Fabric also move the data to bf16?
The XLA team added support for automatic mixed precision (AMP). `XLA:GPU` uses a `GradScaler` and the `autocast` context manager, whereas `XLA:TPU` just uses the latter: https://github.com/pytorch/xla/blob/c9f2d91a234cdaf91f0bbdb044ec94e297ac839a/test/test_train_mp_mnist_amp.py#L143-L147
### Pitch
Integrate `from torch_xla.amp import autocast, GradScaler`
The code would be very similar to the non-XLA AMP plugin: https://github.com/Lightning-AI/lightning/blob/master/src/lightning/fabric/plugins/precision/amp.py
This would likely replace our existing `XLABf16Precision` plugin with an `XLAMixedPrecision` plugin.
### Alternatives
**This was just merged upstream. It's likely very experimental**. I expect it will be released with PyTorch 2.1.
### Additional context
PR on PyTorch: https://github.com/pytorch/pytorch/pull/96370
PR on XLA: https://github.com/pytorch/xla/pull/5161
cc @borda @carmocca @justusschock @awaelchli @JackCaoG @steventk-g @Liyang90
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the existing precision implementations in src/lightning/pytorch/plugins/precision/xlabf16.py and src/lightning/fabric/plugins/precision/xlabf16.py with src/lightning/fabric/plugins/precision/amp.py. Review the linked torch_xla AMP example and determine what integration is needed for TPU and XLA:GPU; done means the existing XLABf16Precision approach is replaced or extended with the requested AMP behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100