facebookresearch / facebookresearch/fairseq2

[LayerSkip] Per-Layer Dropout Rate Configuration

Open
#640 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Describe the solution you would like:**
Would like to enable configuration of a different layer dropout rate for each layer.

**Describe the alternatives you have considered:**
Currently, layer dropout is implemented in fairseq2 as a scalar probability for all layers (check [here](https://github.com/search?q=repo%3Afacebookresearch%2Ffairseq2+layerdrop&type=code)).
We can follow an implementation similar to this [PR in torchtune](https://github.com/pytorch/torchtune/pull/1076) to support linear, exponential, or step configurations for increasing dropout rate acorss layers.

**Additional Context:**
This will enable implementing:
- [Progressive Layer Dropping](https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html): that claims to increase accuracy and speed of training if dropout rate increases across layers linearly
- [LayerSkip](https://arxiv.org/abs/2404.16710): that claims to increase accuracy of early exit layers if dropout rate increaes linearly or exponentially across layers

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.