huggingface / huggingface/lighteval

[BUG] Cannot run math_500/mmlu_pro on models without chat template

Open
#1,089 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
When running math_500/mmlu_pro (and potentially other tasks) on a model without a chat template, I get the following error:

```bash
TypeError: can only concatenate list (not "tuple") to list
```

[at the following line](https://github.com/huggingface/lighteval/blob/58038182c0f8090b4cb023b3400487b56ffdc948/src/lighteval/models/transformers/transformers_model.py#L692).

## To Reproduce

```bash
lighteval accelerate "model_name=HuggingFaceTB/SmolLM2-135M" "mmlu_pro|0"
```

## Expected behavior

Eval to run

## Version info

lighteval==0.13
transformers==4.57.3

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.