huggingface / huggingface/lighteval

[BUG] Num samples not respected

Open
#618 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
num samples metric_option is not respected

## To Reproduce
regardless of options set on task definition and metric settings, it's not generating multiple samples

## Expected behavior

setting
```
metric_options={
"extractive_match": {
"num_samples": 16,
},
```

or

```
# Define tasks
task = LightevalTaskConfig(
num_samples=NUM_SAMPLES, ...
)
```
should respect the setting

## Version info
install
```
conda create -n eval python=3.11
pip install vllm==0.7.2
pip install git+https://github.com/huggingface/lighteval.git#egg=lighteval math-verify==0.5.2
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.