Lightning-AI / Lightning-AI/pytorch-lightning
Guarantee call order for callbacks
@Borda is already working on this.
Since Nov 1, 2021.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
It is often beneficial to have an explicit guarantee that the callbacks will be called in the order that they were added to the callback list. If I understand it right, the order right now is preserved, but it's more of an implementation detail, rather than hard rule.
### Motivation and Pitch
Our use case is quantization + sparsity. During the initial setup we prepare the model for quantization by adding quantization configurations to it. We also prepare the modules for sparsification by adding parametrizations. However, once the training is complete, the merger of the sparse masks and the weights has to happen BEFORE the quantization. Otherwise, the quantization tries to quantize the non-masked tensor.
Another possible use case is when we let the user decide on the order of quantization and masking. If ther user at some point decides to try "quantize -> sparsify" (instead of currently implemented "sparsify -> quantize"), we would want the user to be able to do that.
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning
- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch
- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.