huggingface / huggingface/transformers
Add DistilBERTGeneration comparable to BertGeneration
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
# 🚀 Feature request
I noticed the new `BertGeneration` class, which uses BERT-style models as both encoder and decoder, as well as the more general `EncoderDecoder` class. This is all great stuff! It would also be great to be able to use distilled models. I believe this is possible for the encoder, but for the decoder a language head must be added.
Since DistilBert is implemented as its own model, and not as a BertModel, I don't think it's possible (or at least it's not easy) for the end user to do this. At least not loading pretrained models, since any pretrained model needs to be a type approved by `AutoModelForCausalLM`.
## Motivation
Same motivation as using distilled models in general. Same results at higher speed, this time applied to an `EncoderDecoder` model.
## Your contribution
Happy to be an alpha tester for this feature
Contributor guide
Assessment
This issue has not been assessed yet.