huggingface / huggingface/lighteval

[BUG] Fail to eval on MT-Bench

Open
#858 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
I cannot evaluate model on MT-Bench Task, which was already supported in `src/lighteval/tasks/extended/mt_bench` .

## To Reproduce
```
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,use_chat_template=true,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
lighteval vllm $MODEL_ARGS "extend|mt_bench|0|0" \
--custom-tasks utils/light_eval_tasks.py \
--output-dir $OUTPUT_DIR
```

run the above script will cause the following error when compute metric :
```
TypeError: JudgeLLMMTBench.compute() missing 1 required positional argument: 'docs'
```

Additionally, I noticed that the multi-turn generation which MT-Bench needs does not seem to be functioning correctly, which may also be considered a bug.

## Expected behavior
Successfully eval on MT-Bench

## Version info
The latest version (1.10.1dev0) installed from source.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.