Non-streaming responses leak <think> tags into content for qwen3.5, qwen3.6-moe, qwen3vl and qwen3.5-omni
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 152
- Avg merge
- 4h 14m
- Merged PRs (30d)
- 11
Description
Summary
On the non-streaming OpenAI path, models whose parse_nstream_content does not
split the think block deliver the reasoning inside content instead of
reasoning_content, so the client shows raw <think> tags as the answer. When
generation is truncated mid-thought, the partial thought becomes the entire answer.
Affects qwen3.5, qwen3.6-moe, qwen3vl and qwen3.5-omni — all
reasoning-capable. Reproduced on main at 92f3f13.
Environment
- OS: Ubuntu 26.04.1 LTS (kernel 7.0.0-31-generic)
- NPU driver: NPU FW 1.1.2.64, amdxdna 0.7
flm version:FLM v1.0.6flm validate:
[Linux] Kernel: 7.0.0-31-generic
[Linux] NPU: /dev/accel/accel0 with 8 columns
[Linux] NPU FW Version: 1.1.2.64
[Linux] amdxdna version: 0.7
[Linux] Memlock Limit: infinity
Reproduce
flm serve qwen3.5:9b
curl -s localhost:52625/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen3.5:9b",
"messages": [{"role":"user","content":"What is 5 plus 6? Think briefly."}],
"max_tokens": 400, "think": true
}' | jq '.choices[0].message'
Observed on 92f3f13 — no reasoning_content key, thought in content:
finish=stop reasoning_content=False <think>_in_content=True
content: '<think>\nThinking Process:\n\n1. **Analyze the Request:**\n * '
Same with "max_tokens": 60 (finish=length), where the truncated thought is
the whole answer.
Cause
build_nstream_response takes reasoning_content from parse_nstream_content,
and only some models populate it:
| splits reasoning | does not |
|---|---|
| gemma4_12b, gemma4e, gpt_oss, lfm2, llama3, nanbeige, qwen3 | qwen3_5vl, qwen3_6_moe, qwen3vl, qwen3_5_omni |
The streaming path is unaffected: its parser switches to REASONING on
<think> and stays there, so streamed responses split correctly. Only the
non-streaming path is wrong.
Impact
Agent clients render the thought as the assistant's reply. A truncated thought
reads as the prompt being echoed back ("Thinking Process: 1. Analyze the
Request: The user wants me to ..."), which looks like the model looping.
Fix
PR follows: split the block once in build_nstream_response so every model
reports reasoning the same way, and route an unclosed (truncated) block to
reasoning_content with empty content. No-op for models that already
populate it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating build_nstream_response and parse_nstream_content, then run the provided flm serve and curl reproduction for qwen3.5 with both max_tokens values. Compare non-streaming output with the streaming behavior described in the issue. Done means reasoning is reported in reasoning_content, content is empty for a truncated thought, and models that already split reasoning remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100