NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: NemotronV3 reasoning parser surfaces a trailing think tag as content when the stream ends on it

Open
#19,229 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

LLM API
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • CPU architecture: N/A (reproducible with a pure-Python snippet, no GPU required)
  • GPU: N/A
  • TensorRT-LLM branch: main
  • TensorRT-LLM commit: f7f596b94e
  • OS: N/A

The report concerns NemotronV3ReasoningParser in tensorrt_llm/llmapi/reasoning_parser.py
and affects streaming /v1/chat/completions responses for reasoning_parser: nano-v3.

Who can help?

No response

Information
  • The official example scripts
  • My own modified scripts
Tasks
  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)
Reproduction

DeepSeekR1Parser.parse_delta holds back a delta that is exactly a <think> or
</think> tag (reasoning_end.startswith(delta_text) is true for equality) and waits for
the next delta to decide what it is. When the stream ends on that delta, finish() is
responsible for the withheld buffer. DeepSeekR1Parser.finish() discards a buffer that
holds exactly a tag (reasoning_parser.py:316-320, added in #17157).
NemotronV3ReasoningParser.finish() (reasoning_parser.py:627-655) reimplements the
method without that rule, so the tag is surfaced as model output:

from tensorrt_llm.llmapi.reasoning_parser import ReasoningParserFactory

for kwargs in (None, {"force_nonempty_content": True}):
    parser = ReasoningParserFactory.create_reasoning_parser("nano-v3", kwargs)
    for delta in ["a", "</think>"]:
        parser.parse_delta(delta)
    print(kwargs, parser.finish())

parser = ReasoningParserFactory.create_reasoning_parser("nano-v3", {"enable_thinking": False})
parser.parse_delta("<think>")
print(parser.finish())

Output on main:

None ReasoningParserResult(content='', reasoning_content='</think>')
{'force_nonempty_content': True} ReasoningParserResult(content='a</think>', reasoning_content='')
ReasoningParserResult(content='<think>', reasoning_content='')

</think> is a single token for this model, so any generation that closes its reasoning
and then stops (an empty answer, or a stop right after the tag) ends the stream on exactly
that delta and hits the first two cases on every such request.

Expected behavior

A stream that ends on a complete tag yields the same split as parse() on the joined
text: parse("a</think>") gives reasoning_content="a", content="" (or content="a"
with force_nonempty_content), and parse("<think>") with thinking disabled gives empty
content. The tag is a delimiter and is never part of reasoning_content or content.

actual behavior

The literal </think> is appended to reasoning_content (or, with
force_nonempty_content, to content), and a literal <think> is returned as content
when thinking is disabled. Streaming and non-streaming responses differ for the same
generation.

additional notes
  • DeepSeekR1Parser.finish() is the in-package precedent for the rule; the Nemotron
    override predates it and reimplements the parent's flush without it.
  • The existing test_nano_v3_reasoning_parser_finish covers </think> first and in the
    middle of the stream, but never as the last delta.
  • I have a fix ready (two lines in NemotronV3ReasoningParser.finish(), plus three
    parametrizations in that test) and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorrt_llm/llmapi/reasoning_parser.py, comparing NemotronV3ReasoningParser.finish() with DeepSeekR1Parser.finish(). Run the existing test_nano_v3_reasoning_parser_finish and extend its cases for a stream ending on a complete tag; done means streaming finish() matches parse() without returning delimiter tags as content or reasoning content.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.