facebookresearch / facebookresearch/fairseq2
[LayerSkip] Early Exit Loss
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Describe the solution you would like:**
- Enable the training script to access outputs of intermediate layers
- Modify loss function to incorprate outputs of earlier layers
**Describe the alternatives you have considered:**
Different approaches to implement:
- store output of each layer in a dictionary variable of the model
- return outputs of intermediate layers as an additional return for the forward() function (the disadvantage of this is that it can introduce errors into existing training loops)
- Utilize existing hook mechanism
**Additional Context:**
This is to enable implementing ideas from various papers such as:
- [Depth Adaptive Transformers](https://iclr.cc/virtual_2020/poster_SJg7KhVKPH.html)
- [LayerSkip](https://arxiv.org/abs/2404.16710)
Contributor guide
Assessment
This issue has not been assessed yet.