vllm-project / vllm-project/vllm

[Bug]: RustToolParser streaming can drop buffered text at stream end

Open
#51,549 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
91.9k
Forks
22.3k
PR merge metrics
PR metrics pending

Description

## Your current environment

- vLLM `main` at `04d13b5`
- Reproduced with the Rust tool-parser Python extension in the `vllm/vllm-openai:latest` container
- CPU-only parser reproduction; no model weights or GPU execution are required

## Describe the bug

`RustToolParser` does not finalize the underlying Rust parser when streaming generation ends. A Rust parser may retain an ambiguous partial tool-call marker in its incremental buffer, so the buffered suffix can be silently omitted from streamed content.

For MiniMax M3, the minimal model output is:

```text
plain ]<]minimax[>[[

Contributor guide

Open the contributing guide

Research direction

Start with the streaming adapter's TODO about calling Rust finish(), then trace how DelegatingParser reports finished=True and compare it with the non-streaming adapter's parse_into and finish sequence. Reproduce the CPU-only MiniMax M3 string and verify that stream-end output preserves the buffered suffix, including recovery through reset() when finalization fails for incomplete syntax.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.