huggingface / huggingface/transformers
Request: New LM Adapted checkpoints for T5
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
# 🌟 New LM Adapted checkpoints for T5
## Description
Google released a new set of checkpoints for T5 v1.1. here:
https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511
Especially interesting for most people will be the checkpoints below, as finetuning T5 with a prompt or using T5 for conditional text generation is quite common and these checkpoints promise better performance. The default T5 v1.1 checkpoints have never seen sequences without sentinel tokens.
### LM-Adapted: t5.1.1.lm100k (copied from the readme)
These "LM adapted" models are initialized from t5.1.1 (above) and train for an
additional 100K steps on the LM objective discussed in the [T5 paper][paper].
This adaptation improves the ability of the model to be used for [prompt
tuning](https://arxiv.org/abs/2104.08691).
* **t5.1.1.lm100k.small** (~77 million parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.small](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.small/)
* **t5.1.1.lm100k.base** (~250 million parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.base](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.base/)
* **t5.1.1.lm100k.large** (~800 million parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.large](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.large/)
* **t5.1.1.lm100k.xl** (~3 billion parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.xl](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.xl/)
* **t5.1.1.lm100k.xxl** (~11 billion parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.xxl](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.xxl/)
## Open source status
* [x] the model implementation is available: t5 v1.1. with geglu
* [x] the model weights are available: see links above
* [x] who are the authors: Brian Lester, Rami Al-Rfou, Noah Constant
Contributor guide
Assessment
This issue has not been assessed yet.