`trtllm-eval --max_output_length` is silently discarded on text tasks whose yaml sets `max_gen_toks`

Component: tensorrt_llm/evaluate/lm_eval.py
Verified against: NVIDIA/TensorRT-LLM main (file byte-identical to 1.3.0rc21)
Severity: Medium — a documented CLI flag silently has no effect, with no
warning and no workaround on text tasks

Summary

--max_output_length is advertised as "Maximum generation length" on every
trtllm-eval subcommand. When the lm-eval task yaml defines max_gen_toks, the
CLI value is unconditionally overwritten before generation. Passing a larger
value does nothing.
The guard that exists to prevent this, preserve_caller_max_tokens, is behind
two gates that no text subcommand can satisfy.
Note the clamp itself is defensible — matching the reference harness recipe is
usually correct. The bug is that the flag is offered as a control, silently
ignored, and there is no way to override it for a thinking model.

Root cause

--max_output_length is placed into SamplingParams.max_tokens
(lm_eval.py:715, and again at :1196):

sampling_params = SamplingParams(
    max_tokens=kwargs.pop("max_output_length"),
    truncate_prompt_tokens=kwargs.pop("max_input_length"),
    ...)

_get_sampling_params then overwrites it (lm_eval.py:117-158):

params_mapping = {"temperature": "temperature", "top_p": "top_p",
                  "max_gen_toks": "max_tokens", "until": "stop"}
...
for lm_eval_key, trtllm_key in params_mapping.items():
    value = gen_kwargs.pop(lm_eval_key, None)
    if value is not None and lm_eval_key not in override_keys:
        if (trtllm_key == "max_tokens"
                and self.preserve_caller_max_tokens):   # always False for text
            current = getattr(sampling_params, trtllm_key, None)
            if current is not None and current > value:
                continue
        setattr(sampling_params, trtllm_key, value)     # caller's value destroyed

preserve_caller_max_tokens is False for every text task because of two
independent gates:

1. Not forwarded. evaluate() only passes it to the wrapper for multimodal
   (lm_eval.py:623-627):

   # post_process_fn / preserve_caller_max_tokens only consumed by multimodal.
   if self.MULTIMODAL:
       lm_kwargs["post_process_fn"] = self.post_process_fn
       lm_kwargs["preserve_caller_max_tokens"] = self.preserve_caller_max_tokens
   

   The text LmEvalWrapper keeps its default False (lm_eval.py:65).
2. Not exposed. --preserve_caller_max_tokens is declared on exactly one
   subcommand — MMMU (lm_eval.py:1269).
   Note the asymmetry: --max_input_length maps to truncate_prompt_tokens, which
   is not in params_mapping, so it works as documented. Only the output budget
   is clobbered. --temperature / --top_p have a dedicated escape hatch
   (sampling_override, lm_eval.py:144-146); max_tokens has none for text.

Scope

The clamp applies only when the task yaml defines max_gen_toks — otherwise
value is None and the CLI value survives. Surveyed across the installed
upstream task set plus TRT-LLM's in-repo lm_eval_tasks/:

subcommand	task	yaml max_gen_toks	effect
aime25 / aime26	aime25, aime26	32768	clamped — CLI ignored
longbench_v1	longbench_* subtasks	32 – 512	clamped — CLI ignored
gsm8k	gsm8k	(unset)	CLI value survives — not affected
gpqa_diamond	gpqa_diamond_cot_zeroshot_aa	(unset)	CLI value survives — not affected
mmmu	mmmu_*	512	clamped, but --preserve_caller_max_tokens exists
For LongBench the small caps are appropriate to the tasks (short extractive
answers); they only become a problem for a thinking model whose CoT cannot fit,
and there is no way to raise them.

Observed evidence

DeepSeek-V4-Flash on aime25 (thinking enabled, 4xGB300, TP4/EP4). Of 30
problems, 3 hit the 32768-token cap mid-chain-of-thought — docs 13, 19, 23
produced 91k-102k characters and never emitted </think>, so they scored 0:

doc 13 gold=240 closed_think=False  ~102k chars
doc 19 gold= 60 closed_think=False  ~92k chars
doc 23 gold=735 closed_think=False  ~91k chars

--max_output_length 131072 has no effect; the yaml's 32768 is reapplied per
request. The only way to give those problems more room is to bypass the harness
and drive LLM.generate directly.
Those 3 problems are 10 points of a 30-problem benchmark, lost to an unraisable
cap rather than to model capability.

Suggested fix

Make the existing guard reachable for text tasks — forward the flag
unconditionally (lm_eval.py:623-627):

lm_kwargs["preserve_caller_max_tokens"] = self.preserve_caller_max_tokens
if self.MULTIMODAL:
    lm_kwargs["post_process_fn"] = self.post_process_fn

and expose --preserve_caller_max_tokens on the affected text subcommands
(aime25, aime26, longbench_v1).
Better: treat an explicitly-set --max_output_length the way --temperature
already is. The sampling_override mechanism at lm_eval.py:144-146 is the
existing precedent — adding max_gen_toks to override_keys when the user
explicitly passed the flag would make behavior consistent across all four
sampling parameters.
At minimum, log a warning when the yaml value replaces a caller-supplied
max_tokens, so the clamp is visible rather than silent.

Related

• Companion issue #17020: aime25/aime26 default --chat_template_kwargs to
  {"thinking_budget": 32768}, which DeepSeek-V4 does not read, silently
  disabling thinking (measured 43.33 vs 86.67 on AIME 2025).
• Both share a root pattern: a parameter is accepted, silently ignored, and the
  run completes successfully reporting a wrong number.

Start in tensorrt_llm/evaluate/lm_eval.py by reading evaluate() around lines 623-627 and _get_sampling_params around lines 117-158, then inspect the text subcommand declarations for aime25, aime26, and longbench_v1. Make an explicitly supplied --max_output_length effective when task YAML sets max_gen_toks, and verify the affected text evaluations no longer silently replace the caller’s value.