ROCm / ROCm/FastFlowLM

Non-streaming responses leak <think> tags into content for qwen3.5, qwen3.6-moe, qwen3vl and qwen3.5-omni

Open
#739 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.9k
Forks
152
Avg merge
4h 14m
Merged PRs (30d)
11

Description

Summary

On the non-streaming OpenAI path, models whose parse_nstream_content does not
split the think block deliver the reasoning inside content instead of
reasoning_content, so the client shows raw <think> tags as the answer. When
generation is truncated mid-thought, the partial thought becomes the entire answer.

Affects qwen3.5, qwen3.6-moe, qwen3vl and qwen3.5-omni — all
reasoning-capable. Reproduced on main at 92f3f13.

Environment
  • OS: Ubuntu 26.04.1 LTS (kernel 7.0.0-31-generic)
  • NPU driver: NPU FW 1.1.2.64, amdxdna 0.7
  • flm version: FLM v1.0.6
  • flm validate:
[Linux]  Kernel: 7.0.0-31-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.7
[Linux]  Memlock Limit: infinity
Reproduce
flm serve qwen3.5:9b
curl -s localhost:52625/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3.5:9b",
  "messages": [{"role":"user","content":"What is 5 plus 6? Think briefly."}],
  "max_tokens": 400, "think": true
}' | jq '.choices[0].message'

Observed on 92f3f13 — no reasoning_content key, thought in content:

finish=stop    reasoning_content=False  <think>_in_content=True
content: '<think>\nThinking Process:\n\n1.  **Analyze the Request:**\n    *   '

Same with "max_tokens": 60 (finish=length), where the truncated thought is
the whole answer.

Cause

build_nstream_response takes reasoning_content from parse_nstream_content,
and only some models populate it:

splits reasoning does not
gemma4_12b, gemma4e, gpt_oss, lfm2, llama3, nanbeige, qwen3 qwen3_5vl, qwen3_6_moe, qwen3vl, qwen3_5_omni

The streaming path is unaffected: its parser switches to REASONING on
<think> and stays there, so streamed responses split correctly. Only the
non-streaming path is wrong.

Impact

Agent clients render the thought as the assistant's reply. A truncated thought
reads as the prompt being echoed back ("Thinking Process: 1. Analyze the
Request:
The user wants me to ..."), which looks like the model looping.

Fix

PR follows: split the block once in build_nstream_response so every model
reports reasoning the same way, and route an unclosed (truncated) block to
reasoning_content with empty content. No-op for models that already
populate it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating build_nstream_response and parse_nstream_content, then run the provided flm serve and curl reproduction for qwen3.5 with both max_tokens values. Compare non-streaming output with the streaming behavior described in the issue. Done means reasoning is reported in reasoning_content, content is empty for a truncated thought, and models that already split reasoning remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.