huggingface / huggingface/transformers

Request: New LM Adapted checkpoints for T5

Open
#12,384 0 comments 0 reactions 0 assignees View on GitHub
New model
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

# 🌟 New LM Adapted checkpoints for T5

## Description

Google released a new set of checkpoints for T5 v1.1. here:
https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511

Especially interesting for most people will be the checkpoints below, as finetuning T5 with a prompt or using T5 for conditional text generation is quite common and these checkpoints promise better performance. The default T5 v1.1 checkpoints have never seen sequences without sentinel tokens.

### LM-Adapted: t5.1.1.lm100k (copied from the readme)

These "LM adapted" models are initialized from t5.1.1 (above) and train for an
additional 100K steps on the LM objective discussed in the [T5 paper][paper].
This adaptation improves the ability of the model to be used for [prompt
tuning](https://arxiv.org/abs/2104.08691).

* **t5.1.1.lm100k.small** (~77 million parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.small](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.small/)
* **t5.1.1.lm100k.base** (~250 million parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.base](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.base/)
* **t5.1.1.lm100k.large** (~800 million parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.large](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.large/)
* **t5.1.1.lm100k.xl** (~3 billion parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.xl](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.xl/)
* **t5.1.1.lm100k.xxl** (~11 billion parameters): [gs://t5-data/pretrained_models/t5.1.1.lm100k.xxl](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5.1.1.lm100k.xxl/)

## Open source status

* [x] the model implementation is available: t5 v1.1. with geglu
* [x] the model weights are available: see links above
* [x] who are the authors: Brian Lester, Rami Al-Rfou, Noah Constant

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.