huggingface / huggingface/transformers

Add FAVOR+ / Performer attention

Open
#7,675 46 comments 111 reactions 0 assignees View on GitHub
New model
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

# 🌟 FAVOR+ / Performer attention addition

Are there any plans to add this new attention approximation block to Transformers library?

## Model description
The new attention mechanism with linear time and space complexity was introduced in
_Rethinking Attention with Performers_ [[https://arxiv.org/abs/2009.14794](https://arxiv.org/abs/2009.14794)].
Authors of the paper claim that the new attention mechanism is backward-compatbile with already existing models
> Backwards compatibility with pretrained models is available as a benefit from softmax approximation, via small finetuning (required due to error propagation)

## Open source status

* [x] the model implementation is available: it's an original Trax implementation from Google: https://github.com/google-research/google-research/tree/master/performer/fast_self_attention
* [ ] the model weights are available: probably not required as it's a building block for models rather than fully new architecture
* [x] who are the authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.