[Bug]: DeepSeek V3 streaming tool parser strips markdown fences from content
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
- CPU architecture: N/A (reproducible with a pure-Python snippet, no GPU required)
- GPU: N/A
- TensorRT-LLM branch:
main - TensorRT-LLM commit:
f7f596b94e - OS: N/A
The report concerns tensorrt_llm/serve/tool_parser/deepseekv3_parser.py and affects
streaming /v1/chat/completions responses for tool_parser: deepseek_v3 whenever the
request carries tools.
Who can help?
No response
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
When a streamed delta contains no tool-call markup, DeepSeekV3Parser.parse_streaming_increment
flushes it as content after removing the closing tokens. The list of removed strings
(deepseekv3_parser.py:112) also contains the markdown fence:
for e_token in [self.eot_token, "```", "<|tool▁call▁end|>"]:
normal_text = normal_text.replace(e_token, "")
So every fenced code block in a streamed answer loses its fences. The fence is part of the
tool-call syntax only inside a <|tool▁call▁begin|> ... <|tool▁call▁end|> block, and this
branch is reached only when no start token is present, so the text it sees is model content.
detect_and_parse on the same string returns it unchanged, and the sibling
DeepSeekV31Parser (deepseekv31_parser.py:109) strips only the two closing tokens.
from tensorrt_llm.serve.openai_protocol import ChatCompletionToolsParam, FunctionDefinition
from tensorrt_llm.serve.tool_parser.deepseekv3_parser import DeepSeekV3Parser
tools = [ChatCompletionToolsParam(
type="function",
function=FunctionDefinition(
name="get_weather", description="Get the weather",
parameters={"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]}))]
text = "Here is the code:\n```python\nprint(1)\n```\nDone."
print(repr(DeepSeekV3Parser().parse_streaming_increment(text, tools).normal_text))
print(repr(DeepSeekV3Parser().detect_and_parse(text, tools).normal_text))
Output on main:
'Here is the code:\npython\nprint(1)\n\nDone.'
'Here is the code:\n```python\nprint(1)\n```\nDone.'
Expected behavior
Concatenating the normal_text of every streamed increment reproduces the text that
detect_and_parse returns for the whole response. A delta that carries no tool-call
markup is model content and is streamed verbatim, including markdown fences.
actual behavior
The streaming path removes every ``` from the content, so a client that streams a code
answer receives python\nprint(1)\n instead of a fenced block, while a non-streaming
request for the same output gets the fences.
additional notes
DeepSeekV31Parserand the DSML parsers (deepseek_v32,deepseek_v4) stream the same
text verbatim, sodeepseek_v3is the only parser in the package that alters content
on this path.- The fence is never left over from a completed call: once a call completes, the buffer is
advanced past<|tool▁call▁end|>(deepseekv3_parser.py:188-194), which already
contains the fences. - I have a fix ready (drop
"```"from the list so the parser matchesDeepSeekV31Parser)
and will open a PR referencing this issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tensorrt_llm/serve/tool_parser/deepseekv3_parser.py at parse_streaming_increment and the token list around line 112; compare its behavior with DeepSeekV31Parser and detect_and_parse. Done means streamed content preserves markdown fences and concatenated streaming increments match the non-streaming normal_text for content without tool-call markup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100