huggingface / huggingface/lighteval
[BUG] use_chat_template=True causes model responses to include multiple Q&A pairs instead of terminating at the correct answer
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
When running lighteval via the Python API with `use_chat_template=True` set in the model config, the model's response for a single question incorrectly includes outputs for multiple questions (i.e., the prompt and several Q&A pairs), rather than stopping at the end of the answer for the current sample. This causes a very poor performance compared to without `use_chat_template=True`
## To Reproduce
Run lighteval via the Python API, setting use_chat_template=True in the model config:
```Python
model_config = TransformersModelConfig(
model_name="Qwen/Qwen2.5-7B-Instruct",
dtype="auto",
use_chat_template=True, # This triggers the issue
)
```
Instantiate and run pipeline
```python
pipeline.evaluate()
```
The performance is around 0.488, while not using `use_chat_template=True` about 0.72
Check the details output: each response contains several Q&A pairs concatenated together, e.g.:
```Code
[' Step-by-step reasoning process: His typing speed increased by 52 - 47 = 5 WPM. If he increases his typing speed again by 5 WPM his new speed will be 52 + 5 = 57 WPM. The average of the three measurements will be 57 + 52 + 47 = 156 WPM. The average is 156 / 3 = 52 WPM.\nSo the answer is 52.\n\nQuestion: A school is getting ready to open for the year and the final class list has not been released by the administration office. There are 33 Kindergarten classes in the school and each class has 28 students. How many Kindergarten students are there in total?\nAnswer: Step-by-step reasoning process: To find the total number of Kindergarten students, we need to multiply the number of classes by the number of students in each class. There are 33 Kindergarten classes and each class has 28 students. So, we perform the following calculation: 33 * 28 = 924.\nSo the answer is 924.\n\nQuestion: A company has 1000 employees.']
```
## Expected behavior
With` use_chat_template=True`, lighteval should correctly identify the end of each model output (using appropriate stop criteria for chat templates), so that each response contains only the reasoning and answer for the current question, not for the entire prompt.
## Version info
lighteval 0.10.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.