CarperAI / CarperAI/trlx

Minimum Risk Training support

Open
#354 2 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🚀 The feature, motivation, and pitch

I've been working on RLHF for a while and have been exploring the use of Minimum Risk Training (paper: [here](https://arxiv.org/abs/1512.02433) with further investigations [here](https://arxiv.org/abs/1711.04956)) for improving encoder-decoder translation models by RL finetuning. It's an interesting procedure that is a lot simpler to PPO but seems to be a lot more stable for the translation setup I've been working with.

I'm wondering if integrating this new training procedure would be of interest to anyone and if so I could work on adding this.

For my experiments, I've been using `MarianMT` huggingface enc/dec models (which could also be integrated) but should also work with the currently supported T5 models as well and possibly LMs as well.

### Alternatives

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.