huggingface / huggingface/lighteval
[BUG] Why isn't apply_chat_template called with add_generation_prompt=True in LLM as a judge?
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Is there a particular reason why not using add_generation_prompt=True when tokenizing input for the judge here: https://github.com/huggingface/lighteval/blob/932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f/src/lighteval/metrics/utils/llm_as_judge.py#L304
Without add_generation_prompt=True, the tokenized prompt ends at the user turn (e.g. …<|im_end|>\n) without opening the assistant turn. The judge then has to guess what comes next. Usually it emits the assistant reply fine, but intermittently it continues the conversation (starting a fresh <|im_start|>user … turn) or derails, producing malformed output that fails to parse.
I hit this with Qwen2.5-7B-Instruct as the judge: a fraction of responses came back as user\n… continuations or unrelated text, and setting add_generation_prompt=True fixed them.
Fix is a one-liner:
tokenized = [self.tokenizer.apply_chat_template(p, add_generation_prompt=True) for p in prompt]
Contributor guide
No contributing guide indexed for this repository
Research direction
Open src/lighteval/metrics/utils/llm_as_judge.py around line 304 and inspect how prompts are passed to tokenizer.apply_chat_template. Reproduce the judge prompt with Qwen2.5-7B-Instruct or compare tokenized prompts with and without add_generation_prompt=True; done means the assistant turn is opened and malformed or conversational continuations no longer occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100