huggingface / huggingface/lighteval

[BUG] Evaluation tracker won't save task which relies on using metrics of 2 different types

Open
#950 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
EvaluationTracker.save() will fail at `dataset = Dataset.from_list([asdict(detail) for detail in task_details])` with
```
Exception has occurred: ArrowInvalid (note: full exception trace is shown but execution is paused at: _run_module_as_main)
cannot mix list and non-list, non-null values
```
if the task launched uses metrics requiring both generative and logprobs metrics. (because they don't save lists of the same lengths for ex)

## To Reproduce
```
"name": "launch vllm",
"type": "debugpy",
"request": "launch",
"module": "lighteval",
"args": [
"vllm",
"model_name=Qwen/Qwen3-0.6B,max_num_batched_tokens=100000,max_model_length=38912,generation_parameters={temperature:0.6,top_p:0.95,top_k:20,min_p:0,presence_penalty:1,max_new_tokens:38912},system_prompt='/no_think'",
"lighteval|mmlu_redux_2:security_studies|0", //lighteval|mmlu_redux_2|0|0",
"--max-samples",
"10",
"--save-details",
],
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.