huggingface / huggingface/lighteval

[BUG] use_chat_template=True causes model responses to include multiple Q&A pairs instead of terminating at the correct answer

Open
#867 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
When running lighteval via the Python API with `use_chat_template=True` set in the model config, the model's response for a single question incorrectly includes outputs for multiple questions (i.e., the prompt and several Q&A pairs), rather than stopping at the end of the answer for the current sample. This causes a very poor performance compared to without `use_chat_template=True`

## To Reproduce
Run lighteval via the Python API, setting use_chat_template=True in the model config:
```Python
model_config = TransformersModelConfig(
model_name="Qwen/Qwen2.5-7B-Instruct",
dtype="auto",
use_chat_template=True, # This triggers the issue
)
```
Instantiate and run pipeline
```python
pipeline.evaluate()
```
The performance is around 0.488, while not using `use_chat_template=True` about 0.72
Check the details output: each response contains several Q&A pairs concatenated together, e.g.:
```Code
[' Step-by-step reasoning process: His typing speed increased by 52 - 47 = 5 WPM. If he increases his typing speed again by 5 WPM his new speed will be 52 + 5 = 57 WPM. The average of the three measurements will be 57 + 52 + 47 = 156 WPM. The average is 156 / 3 = 52 WPM.\nSo the answer is 52.\n\nQuestion: A school is getting ready to open for the year and the final class list has not been released by the administration office. There are 33 Kindergarten classes in the school and each class has 28 students. How many Kindergarten students are there in total?\nAnswer: Step-by-step reasoning process: To find the total number of Kindergarten students, we need to multiply the number of classes by the number of students in each class. There are 33 Kindergarten classes and each class has 28 students. So, we perform the following calculation: 33 * 28 = 924.\nSo the answer is 924.\n\nQuestion: A company has 1000 employees.']
```
## Expected behavior
With` use_chat_template=True`, lighteval should correctly identify the end of each model output (using appropriate stop criteria for chat templates), so that each response contains only the reasoning and answer for the current question, not for the entire prompt.

## Version info
lighteval 0.10.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.