NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: DeepSeek V3 streaming tool parser strips markdown fences from content

Open Beginner friendly
#19,225 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • CPU architecture: N/A (reproducible with a pure-Python snippet, no GPU required)
  • GPU: N/A
  • TensorRT-LLM branch: main
  • TensorRT-LLM commit: f7f596b94e
  • OS: N/A

The report concerns tensorrt_llm/serve/tool_parser/deepseekv3_parser.py and affects
streaming /v1/chat/completions responses for tool_parser: deepseek_v3 whenever the
request carries tools.

Who can help?

No response

Information
  • The official example scripts
  • My own modified scripts
Tasks
  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)
Reproduction

When a streamed delta contains no tool-call markup, DeepSeekV3Parser.parse_streaming_increment
flushes it as content after removing the closing tokens. The list of removed strings
(deepseekv3_parser.py:112) also contains the markdown fence:

for e_token in [self.eot_token, "```", "<|tool▁call▁end|>"]:
    normal_text = normal_text.replace(e_token, "")

So every fenced code block in a streamed answer loses its fences. The fence is part of the
tool-call syntax only inside a <|tool▁call▁begin|> ... <|tool▁call▁end|> block, and this
branch is reached only when no start token is present, so the text it sees is model content.
detect_and_parse on the same string returns it unchanged, and the sibling
DeepSeekV31Parser (deepseekv31_parser.py:109) strips only the two closing tokens.

from tensorrt_llm.serve.openai_protocol import ChatCompletionToolsParam, FunctionDefinition
from tensorrt_llm.serve.tool_parser.deepseekv3_parser import DeepSeekV3Parser

tools = [ChatCompletionToolsParam(
    type="function",
    function=FunctionDefinition(
        name="get_weather", description="Get the weather",
        parameters={"type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"]}))]

text = "Here is the code:\n```python\nprint(1)\n```\nDone."

print(repr(DeepSeekV3Parser().parse_streaming_increment(text, tools).normal_text))
print(repr(DeepSeekV3Parser().detect_and_parse(text, tools).normal_text))

Output on main:

'Here is the code:\npython\nprint(1)\n\nDone.'
'Here is the code:\n```python\nprint(1)\n```\nDone.'
Expected behavior

Concatenating the normal_text of every streamed increment reproduces the text that
detect_and_parse returns for the whole response. A delta that carries no tool-call
markup is model content and is streamed verbatim, including markdown fences.

actual behavior

The streaming path removes every ``` from the content, so a client that streams a code
answer receives python\nprint(1)\n instead of a fenced block, while a non-streaming
request for the same output gets the fences.

additional notes
  • DeepSeekV31Parser and the DSML parsers (deepseek_v32, deepseek_v4) stream the same
    text verbatim, so deepseek_v3 is the only parser in the package that alters content
    on this path.
  • The fence is never left over from a completed call: once a call completes, the buffer is
    advanced past <|tool▁call▁end|> (deepseekv3_parser.py:188-194), which already
    contains the fences.
  • I have a fix ready (drop "```" from the list so the parser matches DeepSeekV31Parser)
    and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorrt_llm/serve/tool_parser/deepseekv3_parser.py at parse_streaming_increment and the token list around line 112; compare its behavior with DeepSeekV31Parser and detect_and_parse. Done means streamed content preserves markdown fences and concatenated streaming increments match the non-streaming normal_text for content without tool-call markup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.