ROCm / ROCm/FastFlowLM

Truncated tool call is emitted as a complete one (unterminated heredoc reaches the client)

Open Beginner friendly
#741 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.9k
Forks
152
Avg merge
4h 14m
Merged PRs (30d)
11

Description

Summary

When generation stops inside a <tool_call> block — normally because it hit the
output limit while writing a large argument — the parser takes the remainder of
the text as the block and emits a complete-looking tool call built from partial
input. The client cannot detect this: the arguments JSON is well formed,
because it is rebuilt from the parsed parameters. A shell heredoc cut mid-body
becomes an unterminated cat <<EOF.

Affects qwen3_5vl, qwen3_6_moe and qwen3_5_omni. Reproduced on main at
92f3f13.

Environment
  • OS: Ubuntu 26.04.1 LTS (kernel 7.0.0-31-generic)
  • NPU driver: NPU FW 1.1.2.64, amdxdna 0.7
  • flm version: FLM v1.0.6
  • flm validate:
[Linux]  Kernel: 7.0.0-31-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.7
[Linux]  Memlock Limit: infinity
Reproduce
flm serve qwen3.5:9b
curl -s localhost:52625/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3.5:9b",
  "messages": [{"role":"user","content":"Use the bash tool to write a very long 300-line poem to /tmp/p.txt using a cat heredoc."}],
  "max_tokens": 400,
  "tools": [{"type":"function","function":{"name":"bash","description":"Run a shell command",
             "parameters":{"type":"object","properties":{"command":{"type":"string"}},"required":["command"]}}}],
  "tool_choice": "auto"
}' | jq '.choices[0]'

Observed on 92f3f13:

finish_reason : length
tool_calls    : True
args length   : 1580
args tail     : '...n a sacred soundless gown.\n\nSo let us walk gently on this precious"}'

The heredoc has no closing EOF. A client that executes this hangs waiting for
the terminator.

Cause

parse_nstream_content, e.g. modeling_qwen3_5vl.cpp:493-500:

} else {
    // Unclosed tag — search for </function> fallback
    size_t func_end_pos = response_text.find(func_end_tag, block_content_start);
    if (func_end_pos != std::string::npos) {
        block_end = func_end_pos + func_end_tag.length();
    } else {
        block_end = response_text.length();   // <-- fabricates a complete call
    }

The </function> fallback is reasonable: a block missing only the outer
</tool_call> is still complete. The final else is not — neither closing tag
present means the call was never finished.

Impact

finish_reason is length, which is the only hint, and clients commonly ignore
it when tool_calls is present. The result is a syntactically valid tool call
carrying a truncated command. For file-writing agents that is an unterminated
heredoc; for other tools it is a silently truncated argument.

Fix

PR follows: drop the block when neither closing tag is present, so no tool call
is emitted. The response still carries finish_reason: "length", which is the
honest signal that the client should retry with a larger budget. Complete calls
and the </function> fallback are unaffected.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in modeling_qwen3_5vl.cpp around lines 493-500 and trace parse_nstream_content for blocks lacking both closing tags. Use the provided flm serve and curl reproduction to check the emitted tool_calls and finish_reason. Done means incomplete tool-call blocks are not emitted, while complete calls and the fallback still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.