huggingface / huggingface/transformers

Add DistilBERTGeneration comparable to BertGeneration

Open
#7,397 7 comments 0 reactions 1 assignee Claimed by @patrickvonplaten View on GitHub
Good Second Issue
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

# 🚀 Feature request

I noticed the new `BertGeneration` class, which uses BERT-style models as both encoder and decoder, as well as the more general `EncoderDecoder` class. This is all great stuff! It would also be great to be able to use distilled models. I believe this is possible for the encoder, but for the decoder a language head must be added.

Since DistilBert is implemented as its own model, and not as a BertModel, I don't think it's possible (or at least it's not easy) for the end user to do this. At least not loading pretrained models, since any pretrained model needs to be a type approved by `AutoModelForCausalLM`.

## Motivation

Same motivation as using distilled models in general. Same results at higher speed, this time applied to an `EncoderDecoder` model.

## Your contribution

Happy to be an alpha tester for this feature

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.