facebookresearch / facebookresearch/fairseq2

[LayerSkip] Early Exit Loss

Open
#641 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Describe the solution you would like:**
- Enable the training script to access outputs of intermediate layers
- Modify loss function to incorprate outputs of earlier layers

**Describe the alternatives you have considered:**
Different approaches to implement:
- store output of each layer in a dictionary variable of the model
- return outputs of intermediate layers as an additional return for the forward() function (the disadvantage of this is that it can introduce errors into existing training loops)
- Utilize existing hook mechanism

**Additional Context:**
This is to enable implementing ideas from various papers such as:
- [Depth Adaptive Transformers](https://iclr.cc/virtual_2020/poster_SJg7KhVKPH.html)
- [LayerSkip](https://arxiv.org/abs/2404.16710)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.