CarperAI / CarperAI/trlx

SHP reward model

Open
#336 0 comments 1 reaction 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🚀 The feature, motivation, and pitch

I tried to use SHP + HH but the current RM setup doesn't support it. To support reward models like SHP, instead of adding a value head we need to instead look at the probablities for the preferred response, or the probability for the given response if just one is given, see https://huggingface.co/stanfordnlp/SteamSHP-flan-t5-large for details. This will also open up other ways of zero-shot RM from looking at token probabilities. Should be a relatively easy thing to add.

Maybe we should add a `reward_model` optional field to the config, which we could use to specify either using a triton host, a model run by trlx on one of the ranks, or a model/cpu func run on all ranks. This could also contain the configuration from https://github.com/CarperAI/trlx/pull/322

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.