huggingface / huggingface/transformers
Better Fine-Tuning by Reducing Representational Collapse
Open
Feature request
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
# 🚀 Feature request
Add r3f/r4f to some popular objective functions as suggested by [Armen et. al](https://arxiv.org/abs/2008.03156).
## Motivation
Finetuning is a primary use case of many users of the transformers library.
We can use r3f/r4f to reduce representation collapse by vocabulary inefficiencies.
This is also a relatively cheap feature to implement.
## Your contribution
Understand that this is a relatively new paper with not too much benchmarking done. Will do PR if requested.
Contributor guide
Assessment
This issue has not been assessed yet.