SHP reward model
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 487
- PR merge metrics
- No merged PRs in 30d
Description
### 🚀 The feature, motivation, and pitch
I tried to use SHP + HH but the current RM setup doesn't support it. To support reward models like SHP, instead of adding a value head we need to instead look at the probablities for the preferred response, or the probability for the given response if just one is given, see https://huggingface.co/stanfordnlp/SteamSHP-flan-t5-large for details. This will also open up other ways of zero-shot RM from looking at token probabilities. Should be a relatively easy thing to add.
Maybe we should add a `reward_model` optional field to the config, which we could use to specify either using a triton host, a model run by trlx on one of the ranks, or a model/cpu func run on all ranks. This could also contain the configuration from https://github.com/CarperAI/trlx/pull/322
Contributor guide
Assessment
This issue has not been assessed yet.