huggingface / huggingface/lighteval
RULER benchmark not found
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Hi all,
Thanks for your great work. The [README](https://github.com/huggingface/lighteval?tab=readme-ov-file#-chat-model-evaluation) says lighteval should support RULER benchmark, and I do find relevant PRs (#722, #726, #826 ).
In your previous commit, I can find
> https://github.com/huggingface/lighteval/blob/57f292153d527df81d1bd5e6831f5dfe0fa63279/src/lighteval/tasks/extended/ruler/main.py
but the file was removed. Is there any reason that removing the RULER benchmark? Also, can I easily reuse your previous implementation to perform benchmarking in the current lighteval version (v0.11.0)?
@NathanHB
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.