facebookresearch / facebookresearch/fairseq2
[LayerSkip] Per-Layer Dropout Rate Configuration
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Describe the solution you would like:**
Would like to enable configuration of a different layer dropout rate for each layer.
**Describe the alternatives you have considered:**
Currently, layer dropout is implemented in fairseq2 as a scalar probability for all layers (check [here](https://github.com/search?q=repo%3Afacebookresearch%2Ffairseq2+layerdrop&type=code)).
We can follow an implementation similar to this [PR in torchtune](https://github.com/pytorch/torchtune/pull/1076) to support linear, exponential, or step configurations for increasing dropout rate acorss layers.
**Additional Context:**
This will enable implementing:
- [Progressive Layer Dropping](https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html): that claims to increase accuracy and speed of training if dropout rate increases across layers linearly
- [LayerSkip](https://arxiv.org/abs/2404.16710): that claims to increase accuracy of early exit layers if dropout rate increaes linearly or exponentially across layers
Contributor guide
Assessment
This issue has not been assessed yet.