huggingface / huggingface/lighteval

Error with value of `n`

Open
#737 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Hey there,
while evaluating `Qwen2.5-1.5B` on these tasks `TASKS="lighteval|aime25|0|0,lighteval|gpqa:diamond|0|0,harness|bbh|0|0,lighteval|simpleqa|0|0,extended|lcb|0|0"` using `lighteval vllm`, i get the following error:
```shell
ValueError: n must be 1 when using greedy sampling, got 64
```
I'm using :
```
(lighteval) :~$ pip show lighteval
Name: lighteval
Version: 0.9.2
```

I don't know how or where to pass the value of `n` !
Should it be in the command itself or the yaml config ? and what should be the name ?
Removing the `aime25` task, we get `ValueError: n must be 1 when using greedy sampling, got 16` instead of `64` !

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.