Lightning-AI / Lightning-AI/pytorch-lightning

Guarantee call order for callbacks

Open
#10,260 9 comments 1 reaction 1 assignee View on GitHub

@Borda is already working on this.

Since Nov 1, 2021.

callback question
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature

It is often beneficial to have an explicit guarantee that the callbacks will be called in the order that they were added to the callback list. If I understand it right, the order right now is preserved, but it's more of an implementation detail, rather than hard rule.

### Motivation and Pitch

Our use case is quantization + sparsity. During the initial setup we prepare the model for quantization by adding quantization configurations to it. We also prepare the modules for sparsification by adding parametrizations. However, once the training is complete, the merger of the sparse masks and the weights has to happen BEFORE the quantization. Otherwise, the quantization tries to quantize the non-masked tensor.

Another possible use case is when we let the user decide on the order of quantization and masking. If ther user at some point decides to try "quantize -> sparsify" (instead of currently implemented "sparsify -> quantize"), we would want the user to be able to do that.

______________________________________________________________________

#### If you enjoy Lightning, check out our other projects! ⚡

- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.

- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.