huggingface / huggingface/lighteval
Add HumanEval and HumanEval+
Open
new-task
science-team
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
The HumanEval and HumanEval+ benchmarks are staples for benchmarking code capabilities of base LLMs. It would be nice to include them in `lighteval` so one doesn't have to switch to another framework like BigCode's
References:
* HumanEval: https://github.com/openai/human-eval
* HumanEval+: https://arxiv.org/abs/2305.01210
* Implementation: https://github.com/evalplus/evalplus?tab=readme-ov-file
* BigCode eval harness: https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.