`trtllm-eval aime25`/`aime26` silently disable thinking on DeepSeek-V4, halving reported accuracy

Component: tensorrt_llm/evaluate/lm_eval.py, tensorrt_llm/tokenizer/deepseek_v4/
Version: TensorRT-LLM 1.3.0rc21
Verified against: NVIDIA/TensorRT-LLM main — evaluate/lm_eval.py,
evaluate/interface.py and tokenizer/deepseek_v4/tokenizer.py are all
byte-identical to 1.3.0rc21, so every line number below applies to main. No
existing issue/PR found addressing this.
Severity: High — produces a silently wrong benchmark number with no warning

Summary

The aime25 and aime26 subcommands default --chat_template_kwargs to
'{"thinking_budget": 32768}'. DeepSeek-V4 does not read thinking_budget; its
renderer reads thinking / enable_thinking. The unknown key is silently
ignored, thinking falls back to False, and the benchmark runs in
non-thinking chat mode.
Nothing in the logs or results indicates this happened. The run completes
successfully and reports a plausible-looking score that is roughly half the
model's actual accuracy.

Measured impact

DeepSeek-V4-Flash, AIME 2025, 4×GB300, TP4/EP4, greedy (harness default),
--max_seq_len 36864 --max_output_length 32768, everything else identical:

Reproduction

trtllm-eval --model <DeepSeek-V4-Flash> \
  --tp_size 4 --ep_size 4 --max_batch_size 16 --max_num_tokens 8192 \
  --max_seq_len 36864 --custom_tokenizer deepseek_v4 \
  aime25 --apply_chat_template true --max_output_length 32768

Or without a GPU, directly on the checkpoint tokenizer:

from tensorrt_llm.tokenizer.deepseek_v4 import DeepseekV4Tokenizer
tok = DeepseekV4Tokenizer.from_pretrained("<DeepSeek-V4-Flash>")
msgs = [{"role": "user", "content": "Question: ...\nAnswer:"}]
tok.apply_chat_template(msgs, thinking_budget=32768)[-30:]
# -> '...Answer:<｜Assistant｜></think>'   <-- thinking block PRE-CLOSED
tok.apply_chat_template(msgs, thinking=True)[-30:]
# -> '...Answer:<｜Assistant｜><think>'    <-- correct
tok.apply_chat_template(msgs, thinking=False)[-30:]
# -> '...Answer:<｜Assistant｜></think>'   <-- byte-identical to the default

The default output is byte-identical to explicitly passing {"thinking": false}.

Root cause

tensorrt_llm/tokenizer/deepseek_v4/tokenizer.py:431-437:

def apply_chat_template(self, messages, tools=None, **kwargs):
    tokenize = kwargs.get("tokenize", False)
    thinking = kwargs.get("thinking", False) or kwargs.get("enable_thinking", False)
    thinking_mode = "thinking" if thinking else "chat"
    reasoning_effort = kwargs.get("reasoning_effort")

Every kwarg is read with kwargs.get, so unrecognized keys are silently
dropped. thinking_budget never matches anything.
DeepSeek-V4 uses a hand-written renderer, not a Jinja template (the checkpoint
ships no chat_template in tokenizer_config.json), so the
thinking_budget convention — which works for Jinja templates that reference
that variable — has no effect here.
The defaults are at lm_eval.py:1630 (aime25) and lm_eval.py:1513 (aime26).
These are the only two subcommands with a non-None default; the others merely
mention thinking_budget in help text.
get_chat_template_kwargs (evaluate/interface.py:33-48) does not catch this
either — its guard only fires when the tokenizer's chat_template is a string
containing "enable_thinking", and DeepSeek-V4 has no template at all.
Note the tokenizer is selected automatically from model_type: deepseek_v4
(tokenizer/tokenizer.py:44), so this affects users who never pass
--custom_tokenizer.

Suggested fix

Two changes, ideally both:
1. Accept thinking_budget in the V4 renderer. The existing help text
already promises the semantic ("set to 0 to disable thinking"), so honoring it
makes the documented contract true:

thinking = bool(kwargs.get("thinking", False)
                or kwargs.get("enable_thinking", False)
                or (kwargs.get("thinking_budget") or 0) > 0)

2. Warn on unrecognized chat_template_kwargs keys for the custom
(non-Jinja) tokenizers. The silent no-op is the actual hazard here — a wrong
number that looks right is worse than an error. A logger.warning listing
ignored keys would have surfaced this immediately.

Related, non-blocking observations

• 3/30 thinking-mode responses hit the 32768-token cap mid-CoT (91k–102k
  chars, never emitted </think>) and scored 0. So 86.67 is itself a floor for
  this model; the upstream aime25.yaml max_gen_toks: 32768 is binding.
• --output_path is treated as a directory (lm_eval.py:592-594 calls
  mkdir(parents=True) then writes <path>/samples_<task>.json). Passing a
  .json filename creates a directory with that name. Worth a docstring note.
• Scores in the saved JSON are already normalized to 0–100 (lm_eval.py:640-643
  multiplies before save_results), which is easy to double-scale.

Start with the defaults at evaluate/lm_eval.py:1513 and :1630, then read get_chat_template_kwargs in evaluate/interface.py and apply_chat_template in tokenizer/deepseek_v4/tokenizer.py:431-437. Use the checkpoint-tokenizer reproduction and AIME command to verify that thinking is enabled for DeepSeek-V4 and that unrecognized chat-template keys produce a warning; done means the default no longer silently reports non-thinking accuracy.