CarperAI / CarperAI/trlx

Best-of-n sampling for rollouts

Open
#166 1 comment 0 reactions 1 assignee Claimed by @PhungVanDuy View on GitHub
feature request
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🚀 The feature, motivation, and pitch

Currently PPO does not use best-of-n for sampling for exploration. I think that its inclusion would improve sample efficiency massively, especially because inference is relatively cheap when compared to backwards passes.

Currently hugging face does not have best-of-n sampling built in, so we would need to add it as an option. I have best-of-n sampling implemented here for chatbots, [see here](https://github.com/LouisCastricato/Audience-Comprehension/blob/58b342cb4dca3724ae77c57a5df9cc4167adbe0d/audience/model/utils.py#L18). It should be relatively easy to port over to trlx.

### Alternatives

_No response_

### Additional context

honk

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.