huggingface / huggingface/lighteval

[BUG] Breaking change for Hellaswag/Piqa

Open
#1,090 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
In the most recent versions of lighteval, there was a breaking change for some tasks, like Hellaswag and Piqa.
The default behaviour went from something similar to what lm_eval does (like using `loglikelihood_acc` as metric) to instead using `exact_match`.

Until version `0.9.2`, this was not the case.

Furthermore, in version `0.13.0` the [prompt functions](https://github.com/huggingface/lighteval/blob/b77c6b251e0510b3e05447119a96c10c6e0fa8d9/src/lighteval/tasks/default_prompts.py#L987-L995) and utilities needed to reproduce the `lm_eval`-like behaviour have been removed

## To Reproduce
Run lighteval on any model testing against `hellaswag` or `piqa`.

## Expected behavior
It would be nice to have the possibility to run these tasks following a similar setup used in lm_eval as it was possible in previous versions of lighteval.

## Version info
lighteval==0.13.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.