huggingface / huggingface/lighteval
Error with value of `n`
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Hey there,
while evaluating `Qwen2.5-1.5B` on these tasks `TASKS="lighteval|aime25|0|0,lighteval|gpqa:diamond|0|0,harness|bbh|0|0,lighteval|simpleqa|0|0,extended|lcb|0|0"` using `lighteval vllm`, i get the following error:
```shell
ValueError: n must be 1 when using greedy sampling, got 64
```
I'm using :
```
(lighteval) :~$ pip show lighteval
Name: lighteval
Version: 0.9.2
```
I don't know how or where to pass the value of `n` !
Should it be in the command itself or the yaml config ? and what should be the name ?
Removing the `aime25` task, we get `ValueError: n must be 1 when using greedy sampling, got 16` instead of `64` !
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.