huggingface / huggingface/lighteval

RULER benchmark not found

Open
#1,001 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Hi all,

Thanks for your great work. The [README](https://github.com/huggingface/lighteval?tab=readme-ov-file#-chat-model-evaluation) says lighteval should support RULER benchmark, and I do find relevant PRs (#722, #726, #826 ).

In your previous commit, I can find
> https://github.com/huggingface/lighteval/blob/57f292153d527df81d1bd5e6831f5dfe0fa63279/src/lighteval/tasks/extended/ruler/main.py

but the file was removed. Is there any reason that removing the RULER benchmark? Also, can I easily reuse your previous implementation to perform benchmarking in the current lighteval version (v0.11.0)?

@NathanHB

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.