Best-of-n sampling for rollouts
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 487
- PR merge metrics
- No merged PRs in 30d
Description
### 🚀 The feature, motivation, and pitch
Currently PPO does not use best-of-n for sampling for exploration. I think that its inclusion would improve sample efficiency massively, especially because inference is relatively cheap when compared to backwards passes.
Currently hugging face does not have best-of-n sampling built in, so we would need to add it as an option. I have best-of-n sampling implemented here for chatbots, [see here](https://github.com/LouisCastricato/Audience-Comprehension/blob/58b342cb4dca3724ae77c57a5df9cc4167adbe0d/audience/model/utils.py#L18). It should be relatively easy to port over to trlx.
### Alternatives
_No response_
### Additional context
honk
Contributor guide
Assessment
This issue has not been assessed yet.