huggingface / huggingface/lighteval
[BUG] Breaking change for Hellaswag/Piqa
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
In the most recent versions of lighteval, there was a breaking change for some tasks, like Hellaswag and Piqa.
The default behaviour went from something similar to what lm_eval does (like using `loglikelihood_acc` as metric) to instead using `exact_match`.
Until version `0.9.2`, this was not the case.
Furthermore, in version `0.13.0` the [prompt functions](https://github.com/huggingface/lighteval/blob/b77c6b251e0510b3e05447119a96c10c6e0fa8d9/src/lighteval/tasks/default_prompts.py#L987-L995) and utilities needed to reproduce the `lm_eval`-like behaviour have been removed
## To Reproduce
Run lighteval on any model testing against `hellaswag` or `piqa`.
## Expected behavior
It would be nice to have the possibility to run these tasks following a similar setup used in lm_eval as it was possible in previous versions of lighteval.
## Version info
lighteval==0.13.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.