mudler / mudler/vllm.cpp

fix(TOOLS-STREAMING-PARSER): bare MESSAGE_HEADER text is dropped instead of emitted

Open
#2,637 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: TOOLS-STREAMING-PARSER

Found by re-deriving upstream 629a938a92 (vllm#50403) against this tree in
#2632, wave PORTQ-1.

The defect

Text a model emits while the streaming parser is in MESSAGE_HEADER is dropped
and never reaches the client. The engine has no arm for that state and no buffer
to hold it, so the text is discarded at four separate points:

  • src/vllm/parser/engine/streaming_parser_engine.cpp:297-305emit_for_state
    has no MESSAGE_HEADER arm; it falls through to content_event_for and
    returns {} when that misses.
  • include/vllm/parser/engine/parser_engine_config.h:73-78 — the content_events
    default map carries CONTENT/REASONING/TOOL_NAME/TOOL_ARGS only, so
    MESSAGE_HEADER always misses.
  • src/vllm/parser/engine/streaming_parser_engine.cpp:221-222finish()
    leaves MESSAGE_HEADER by setting state_ = CONTENT and emitting nothing.
  • src/vllm/parser/engine/configs.cpp:1002
    c.transitions[{S::MESSAGE_HEADER, end}] = Transition(S::CONTENT, {});
    (and :985, :990, :991 for the other MESSAGE_HEADER exits), the empty
    event tuple upstream's fix changes.

grep -rn 'message_header_buffer\|MessageHeaderBuffer' include src returns
nothing.

Upstream

vllm/parser/engine/streaming_parser_engine.py @ 629a938a92 (a forward
reference past the pin 5559679229): :359-361 buffers MESSAGE_HEADER text
into _message_header_buffer instead of dropping it, :388-425 emits the buffer
as a TEXT_CHUNK on the transition out and suppresses empty chunks, :274-282
re-emits it when finish() leaves MESSAGE_HEADER, and :188/:324 clear it on
reset and on the skip-tool-parsing path. parser/inkling.py:270-271 changes the
(MESSAGE_HEADER, end) transition's events from () to (TEXT_CHUNK,).

This is user-visible: a bare Inkling response with no thinking block loses its
text.

Size

~35-45 lines in src/vllm/parser/engine/streaming_parser_engine.cpp, one member
in its header, one line in src/vllm/parser/engine/configs.cpp, plus a red-first
streaming test.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with emit_for_state and finish() in src/vllm/parser/engine/streaming_parser_engine.cpp, then inspect the content_events defaults in include/vllm/parser/engine/parser_engine_config.h and MESSAGE_HEADER transitions in src/vllm/parser/engine/configs.cpp. Compare the referenced upstream behavior and add the red-first streaming test mentioned in the issue; done means bare MESSAGE_HEADER text reaches the client, including on finish, without empty chunks.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.