mudler / mudler/vllm.cpp

fix(TOOLS-PARSER-BREADTH): the HY-V3 streaming parser drops leading content and every parallel call after the first

Open
#2,727 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: TOOLS-PARSER-BREADTH (.agents/engine-matrix.md:166). hy_v3 is not named in the row's prose, but the row covers "the remaining --tool-call-parser dialects" and the parser-name set that src/vllm/entrypoints/openai/tool_parsers/abstract.cpp:296 enumerates, of which hy_v3 is one. Row resolved by line number before filing.

Found by wave PORTQ-7 (#2717) re-deriving PORT-NOW entry [258], upstream f18e29834b vllm#53965. Nothing was executed.

The defect

Two things are silently lost by the HY-V3 streaming tool parser:

  1. Content preceding <tool_calls> in a delta is dropped whenever the buffer does not yet yield a complete tool call.
  2. A second parallel tool call delivered in the same streaming delta is never recovered — only the first is emitted.

This tree carries the parser and carries the exact pre-commit shape at all five sites the commit touches. src/vllm/entrypoints/openai/tool_parsers/hy_v3.cpp is 625 lines, with include/vllm/entrypoints/openai/tool_parsers/hy_v3.h, registered at abstract.cpp:189, listed at :296, auto-detected at detect.cpp:101 ({"hy_v3", "<tool_calls"}), and tested through tests/vllm/entrypoints/openai/tool_parsers/test_hy_v3.cpp, wired at tests/CMakeLists.txt:2243.

The five sites:

  1. hy_v3.cpp:386-391 — the early return the commit deletes: if (!before.empty()) { DeltaMessage msg; msg.content = before; return msg; }, instead of capturing before as a content_delta carried forward.
  2. hy_v3.cpp:407-410if (start_idx == npos && !streaming_tool_name_) { buffer_.clear(); return std::nullopt; }, with no content emission before the drop.
  3. hy_v3.cpp:416-418if (sep_idx == npos) { buffer_ = cur_text.substr(start_idx); return std::nullopt; }, same.
  4. hy_v3.cpp:427-434 — builds name_delta with no .content assignment.
  5. hy_v3.cpp:594-622 — the tail of ExtractStreamingIncremental carries the three pre-commit branches and no _drain_calls equivalent, so nothing ever pulls a second complete <tool_call>...</tool_call> span out of buffer_.

grep -rni "drain" over hy_v3.cpp, hy_v3.h and test_hy_v3.cpp returns rc 1 with no hits; the positive control buffer_ over hy_v3.cpp through the identical form returns 11. grep -rni "content_delta" src/ include/ tests/ returns 10 hits, none of them in any hy_v3 file — which is its own positive control: the needle exists in this tree, just not in this parser.

The commit is the whole distance

git show 5559679229:vllm/tool_parsers/hy_v3_tool_parser.py:421-455 matches this tree line-for-line in structure — the same early return, the same two bare return Nones. Not a pre-pin hole. (hy_v3.h:1 cites "@ e24d1b24" rather than the pin, but the pin blob and this tree agree on every line the commit touches, so the older anchor does not change the reading.)

git log --oneline 5559679229..e126687a9a -- vllm/tool_parsers/hy_v3_tool_parser.py returns exactly one SHA, this one. Nothing in range narrows or withdraws it.

What is missing

Thread an std::optional<std::string> content_delta from extract_tool_calls_streaming (hy_v3.cpp:368) through ExtractStreamingIncremental (:446), emit it at the two early-out points, and hang it on name_delta and on the combined name+args delta. Add a DrainCalls(request) helper that repeatedly pulls complete <tool_call>...</tool_call> spans out of buffer_, appends them to prev_tool_call_arr / streamed_args_for_tool, and extends the returned delta's tool_calls.

Note the signature change: ExtractStreamingIncremental currently takes only (name_delta, request), so the content delta has to arrive as a parameter or a member.

Size

~60-85 product lines in one file — DrainCalls is roughly 30 in C++, plus ~12 threading edits across the five sites, plus the signature change. Tests must be authored, not ported: git show --stat f18e29834b lists one file and no test. Budget ~100-150 lines in test_hy_v3.cpp for (a) content preceding <tool_calls> in the same delta as an incomplete call, and (b) two complete calls in one delta.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vllm/entrypoints/openai/tool_parsers/hy_v3.cpp, especially ExtractStreamingIncremental and the five listed sites, then review tests/vllm/entrypoints/openai/tool_parsers/test_hy_v3.cpp and its tests/CMakeLists.txt wiring. Run the existing HY-V3 parser tests first; done means preserving leading content and emitting all complete parallel calls, with regression tests for both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.