NVIDIA / NVIDIA/TensorRT-LLM

MODEL_TYPE_TO_TOOL_PARSER maps qwen3_5/qwen3_5_moe to a parser whose format the template does not emit

Open Beginner friendly
#18,084 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

LLM API
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • TensorRT-LLM: 1.3.0rc23
  • GPU: NVIDIA GeForce RTX 5090 (sm_120), driver 595.71.05
  • Backend: PyTorch; single GPU, no TP/PP
  • Models: Qwen/Qwen3.8-27B (qwen3_5 family); same class of mismatch previously observed on Qwen3.6-35B-A3B
Who can help?

No response

Reproduction

MODEL_TYPE_TO_TOOL_PARSER (tensorrt_llm/serve/tool_parser/tool_parser_factory.py) maps model
types qwen3_5 and qwen3_5_moe to the qwen3 parser, which expects the JSON tool-call form.

The chat template for these models emits the XML-ish form instead:

<tool_call><function=name><parameter=k>v</parameter></function></tool_call>

Serve with automatic tool-parser selection and issue a request with tools defined. Measured on
a four-call scenario:

  • with the auto-selected qwen3 parser: 0 of 4 tool calls parsed
  • with --tool_parser qwen3_coder: 4 of 4 parsed, arguments correct including an optional enum
Expected behavior

Automatic selection maps a model family to a parser that can parse the format that family's template emits.

actual behavior

tool_calls comes back [] and content is emptied, with HTTP 200 and no error or warning. The response is indistinguishable from a model that chose not to call a tool, so a caller cannot detect the failure.

additional notes

--tool_parser qwen3_coder parses this format correctly on the same build and model, so the
parser exists and only the mapping is wrong.

Related but distinct: #17917 reports that a parser which matches its marker yet extracts no calls
returns 200 with no diagnostic — that is the silence; this issue is the mapping that triggers
it here. Fixing either alone leaves the other. #17574 and #17740 concern the streaming parser.

Questions for maintainers

  1. Should qwen3_5/qwen3_5_moe map to qwen3_coder, or is the correct discriminator the
    template content rather than the model type?
  2. Are other entries in MODEL_TYPE_TO_TOOL_PARSER verified by effect against their templates?
    The failure mode here is silent, so an unverified entry looks identical to a working one.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorrt_llm/serve/tool_parser/tool_parser_factory.py and compare the qwen3 and qwen3_coder parser mappings with the tool-call format emitted by the qwen3_5 templates. Reproduce automatic parser selection with tools defined and verify that qwen3_5 and qwen3_5_moe produce parsed tool calls with their arguments rather than empty content and tool_calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.