huggingface / huggingface/lighteval
[BUG] Cannot run math_500/mmlu_pro on models without chat template
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
When running math_500/mmlu_pro (and potentially other tasks) on a model without a chat template, I get the following error:
```bash
TypeError: can only concatenate list (not "tuple") to list
```
[at the following line](https://github.com/huggingface/lighteval/blob/58038182c0f8090b4cb023b3400487b56ffdc948/src/lighteval/models/transformers/transformers_model.py#L692).
## To Reproduce
```bash
lighteval accelerate "model_name=HuggingFaceTB/SmolLM2-135M" "mmlu_pro|0"
```
## Expected behavior
Eval to run
## Version info
lighteval==0.13
transformers==4.57.3
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.