huggingface / huggingface/lighteval
[BUG] Fail to eval on MT-Bench
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
I cannot evaluate model on MT-Bench Task, which was already supported in `src/lighteval/tasks/extended/mt_bench` .
## To Reproduce
```
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,use_chat_template=true,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
lighteval vllm $MODEL_ARGS "extend|mt_bench|0|0" \
--custom-tasks utils/light_eval_tasks.py \
--output-dir $OUTPUT_DIR
```
run the above script will cause the following error when compute metric :
```
TypeError: JudgeLLMMTBench.compute() missing 1 required positional argument: 'docs'
```
Additionally, I noticed that the multi-turn generation which MT-Bench needs does not seem to be functioning correctly, which may also be considered a bug.
## Expected behavior
Successfully eval on MT-Bench
## Version info
The latest version (1.10.1dev0) installed from source.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.