Reasoning-parser auto-selection misclassifies reasoning-at-start templates, silently emptying reasoning_content
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
- TensorRT-LLM: 1.3.0rc23
- GPU: NVIDIA GeForce RTX 5090 (sm_120), driver 595.71.05
- Backend: PyTorch; single GPU, no TP/PP
- Model:
Qwen/Qwen3.8-27B(also reproduced with the NVFP4 conversion of the same model)
Who can help?
No response
Reproduction
_resolve_qwen3_reasoning_parser (tensorrt_llm/llmapi/reasoning_parser.py) classifies a chat
template as "hybrid" when the template contains an enable_thinking toggle, and selects the
qwen3 parser, which expects a matched <think>...</think> pair.
Qwen3.8's chat template injects <think>\n into the generation prompt whenever
enable_thinking is undefined — which is the default. The model's output therefore begins
already inside the reasoning span: it contains a closing </think> and no opening <think>.
Serve with automatic parser selection and send any prompt:
trtllm-serve <qwen3.8-27b> --backend pytorch # no --reasoning_parser
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"What is 17*23?"}]}'
The three-way classification has no case for "template has a toggle, but injects the opening
marker when that toggle is undefined".
Expected behavior
Automatic selection picks a parser registered reasoning_at_start=True, so reasoning_content carries the chain of thought and content carries only the answer.
actual behavior
The qwen3 parser finds no opening marker, splits nothing, and returns reasoning_content empty while raw chain-of-thought is emitted into content. No warning is raised, so a caller that renders content shows the model's private reasoning to the end user.
additional notes
Explicitly selecting --reasoning_parser deepseek-r1, which is registered
reasoning_at_start=True, produces correct separation on the same build and model. That is the
workaround we run in production, and it is also the evidence that the parser set is adequate and
only the selection is wrong.
This is a selection defect and is distinct from two nearby reports: #17916 concerns
DeepSeekR1Parser ignoring a per-request enable_thinking=False, and #17296 concerns streaming
and non-streaming parsers disagreeing. Both assume the correct parser was chosen.
Questions for maintainers
- Should the classifier gain a fourth case, or should templates that inject an opening marker be
detected directly (e.g. by rendering the generation prompt and inspecting it)? - Is a warning appropriate when a selected parser produces an empty
reasoning_contentfor every
response in a session? The silence is what makes this reach production.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tensorrt_llm/llmapi/reasoning_parser.py at _resolve_qwen3_reasoning_parser and trace how templates with enable_thinking are classified. Reproduce the Qwen3.8 behavior with automatic selection, then verify that the selected reasoning_at_start parser separates reasoning_content from content without requiring an explicit parser option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai-infra-agents, backend-api-design
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 66/100