facebookresearch / facebookresearch/fairseq2
[LayerSkip] Self-Speculative Decoding
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Describe the solution you would like:**
Implement self-speculative decoding as described in this [paper](https://arxiv.org/abs/2404.16710) where the earlier layers act as the draft stage and remaining layers act as the verification stage.
**Describe the alternatives you have considered:**
There are different options to implement that:
- Implement regular Speculative Decoding where the draft stage is a separate model, and then Self-Speculative Decoding could be implemented by providing a subset of the layers as the draft model (e.g., this [implementation](https://github.com/pytorch-labs/gpt-fast/blob/091515ab5b06f91c0d6a3b92f9c27463f738cc9b/generate.py#L87C1-L136C1))
- If we use this setup, we can add some flags to inform earlier layers if they are running the draft stage or verification stage
- Directly implement Self-Speculative Decoding as done [here](https://github.com/pytorch-labs/gpt-fast/compare/main...LayerSkip)
**Additional Context:**
- Speculative Decoding was first proposed in [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192)
- Another variant of self-speculative decoding where the draft stage is a subset of the layers of the main model is presented in [Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding](https://arxiv.org/abs/2309.08168)
Contributor guide
Assessment
This issue has not been assessed yet.