huggingface / huggingface/lighteval
`apply_target_perplexity_metric` pops only the first response
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Hi there.
`apply_target_perplexity_metric` seems to pop only the first related response not all of them.
https://github.com/huggingface/lighteval/blob/ad42e43bcc3bd50fdba68936999bf553bf53b9e4/src/lighteval/metrics/__init__.py#L31-L36
While its related responses are as much as the number of gold choices.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.