huggingface / huggingface/lighteval

[BUG] transformers backend overrides the model's generation_config.eos_token_id, so chat models whose turn terminator differs from tokenizer.eos never stop (Gemma pads to max_new_tokens)

Open Beginner friendly
#1,278 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

For generative tasks with `use_chat_template=True`, the transformers backend forces `eos_token_id = tokenizer.eos_token_id` and relies on it as the only stopping criterion. Models whose chat-turn terminator is a different token than `tokenizer.eos` therefore never stop: they emit their turn-end token, it is ignored, and generation runs to `max_new_tokens`.

Concrete case, `google/gemma-4-E2B-it`: `tokenizer.eos_token` is `` (id 1), but the model ends chat turns with token 106 and declares `generation_config.eos_token_id = [1, 106, 50]`. With a chain-of-thought MMLU task (`generation_size` 7168), every single generation ran to the cap: 63-95% of the returned tokens were token 106 repeated, real content was only ~400-2700 tokens.

## Where it happens

`src/lighteval/models/transformers/transformers_model.py` (current main):

- L808-812: `generation_config.update(..., eos_token_id=self.tokenizer.eos_token_id, ...)` clobbers whatever `eos_token_id` the model's own generation config declares (the dict copied at L808 is overwritten by the update).
- L554-556 and L693-695: both generation paths assume "for chat models, generation stops with EOS token" (`stop_tokens = []` / `[self.tokenizer.eos_token]`), which bakes in the same assumption. This dates back to #115.

Task-level `stop_sequence` cannot work around it for chat models, since it is dropped on that path.

## Measured impact (Gemma-4-E2B-it, MMLU CoT, one item, RTX PRO 6000, bf16)

| | before | after the 1-line fix |
|---|---|---|
| tokens_generated | 7168 (= cap) | 2654 (stopped on its own) |
| token-106 padding | 6774 | 1 |
| latency / item | 145 s | 55 s |
| extracted answer | correct | correct (unchanged) |

Besides ~3x wall-clock waste, this silently corrupts `tokens_generated` as a measurement (it tracks the cap, not the model), which matters for any analysis using generation length.

## Suggested fix (validated above)

Prefer the model's declared terminators, falling back to the tokenizer's:

```python
eos_token_id=(self.model.generation_config.eos_token_id or self.tokenizer.eos_token_id),
```

Happy to open a PR with this change.

## Related

Same footgun class (Gemma turn terminator != ``) as huggingface/transformers#38182 and unslothai/unsloth#5386, but this instance is in lighteval's own generate call.

## Version

Reproduced against lighteval main (verified still present at HEAD as of 2026-07-02; originally found at 3fd1526, 2026-06-26).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/lighteval/models/transformers/transformers_model.py, especially the generation-config update around L808-812 and the chat generation paths around L554-556 and L693-695. Reproduce the Gemma chat-generation case or inspect the existing generation flow; done means the model's declared terminators are preserved, tokenizer fallback remains available, and generation stops before max_new_tokens when the model emits its turn terminator.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.