huggingface / huggingface/open-muse
Relative Positional Embeddings
Open
- Dominant language
- Python
- Stars
- 359
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
Typically, we try to remove positional embeddings so that the model can work with different resolutions but I heard a lot of people suggesting relative positional embeddings https://github.com/ofirpress/attention_with_linear_biases
It might be worth a try if we get time
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.