huggingface / huggingface/lighteval
[BUG] transformers backend overrides the model's generation_config.eos_token_id, so chat models whose turn terminator differs from tokenizer.eos never stop (Gemma pads to max_new_tokens)
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
For generative tasks with `use_chat_template=True`, the transformers backend forces `eos_token_id = tokenizer.eos_token_id` and relies on it as the only stopping criterion. Models whose chat-turn terminator is a different token than `tokenizer.eos` therefore never stop: they emit their turn-end token, it is ignored, and generation runs to `max_new_tokens`.
Concrete case, `google/gemma-4-E2B-it`: `tokenizer.eos_token` is `` (id 1), but the model ends chat turns with token 106 and declares `generation_config.eos_token_id = [1, 106, 50]`. With a chain-of-thought MMLU task (`generation_size` 7168), every single generation ran to the cap: 63-95% of the returned tokens were token 106 repeated, real content was only ~400-2700 tokens.
## Where it happens
`src/lighteval/models/transformers/transformers_model.py` (current main):
- L808-812: `generation_config.update(..., eos_token_id=self.tokenizer.eos_token_id, ...)` clobbers whatever `eos_token_id` the model's own generation config declares (the dict copied at L808 is overwritten by the update).
- L554-556 and L693-695: both generation paths assume "for chat models, generation stops with EOS token" (`stop_tokens = []` / `[self.tokenizer.eos_token]`), which bakes in the same assumption. This dates back to #115.
Task-level `stop_sequence` cannot work around it for chat models, since it is dropped on that path.
## Measured impact (Gemma-4-E2B-it, MMLU CoT, one item, RTX PRO 6000, bf16)
| | before | after the 1-line fix |
|---|---|---|
| tokens_generated | 7168 (= cap) | 2654 (stopped on its own) |
| token-106 padding | 6774 | 1 |
| latency / item | 145 s | 55 s |
| extracted answer | correct | correct (unchanged) |
Besides ~3x wall-clock waste, this silently corrupts `tokens_generated` as a measurement (it tracks the cap, not the model), which matters for any analysis using generation length.
## Suggested fix (validated above)
Prefer the model's declared terminators, falling back to the tokenizer's:
```python
eos_token_id=(self.model.generation_config.eos_token_id or self.tokenizer.eos_token_id),
```
Happy to open a PR with this change.
## Related
Same footgun class (Gemma turn terminator != ``) as huggingface/transformers#38182 and unslothai/unsloth#5386, but this instance is in lighteval's own generate call.
## Version
Reproduced against lighteval main (verified still present at HEAD as of 2026-07-02; originally found at 3fd1526, 2026-06-26).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/lighteval/models/transformers/transformers_model.py, especially the generation-config update around L808-812 and the chat generation paths around L554-556 and L693-695. Reproduce the Gemma chat-generation case or inspect the existing generation flow; done means the model's declared terminators are preserved, tokenizer fallback remains available, and generation stops before max_new_tokens when the model emits its turn terminator.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100