huggingface / huggingface/transformers

Better Fine-Tuning by Reducing Representational Collapse

Open
#10,201 2 comments 0 reactions 0 assignees View on GitHub
Feature request
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

# 🚀 Feature request

Add r3f/r4f to some popular objective functions as suggested by [Armen et. al](https://arxiv.org/abs/2008.03156).

## Motivation

Finetuning is a primary use case of many users of the transformers library.
We can use r3f/r4f to reduce representation collapse by vocabulary inefficiencies.
This is also a relatively cheap feature to implement.

## Your contribution

Understand that this is a relatively new paper with not too much benchmarking done. Will do PR if requested.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.