huggingface / huggingface/lighteval
[EVAL] Add RewardBench
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Evaluation short description
- Why is this evaluation interesting?
RewardBench is perhaps the _only_ evaluation suite to provide broad coverage of the strengths and weaknesses of reward models across domains like reasoning, chat, and safety. It would be great to have this in `lighteval` so one can have a unified evaluation suite and not have to hop across different repos and dependencies.
- How used is it in the community?
RewardBench is the de facto leaderboard for comparing reward models and is widely used by the post-training subset of the community for advanced methods like RL, rejection sampling, and others.
## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2403.13787
- Github url: https://github.com/allenai/reward-bench
- Dataset url: https://huggingface.co/datasets/allenai/reward-bench
**Note:** this eval is not a typical LLM eval since it relies on sequence classification. I'm not sure if that is out of scope for `lighteval`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.