vllm-project / vllm-project/agentic-api
refactor: extract StreamRelay and bound client delivery
@LOGO127 is already working on this.
Since Sep 14, 2026.
- Dominant language
- Rust
- Stars
- 284
- Forks
- 74
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 93
Description
ingestion and client emission currently share control flow, while the final client-facing channel is unbounded. bounding proposed worker queues alone would still allow slow-client buffering to grow. we should isolate emission and make backpressure reach the client boundary, independently of any worker-placement decision.
proposed second slice of #241, building on the synchronous ingestion boundary in #243. this is a suggested scope; feel free to revise, combine, or remove it.
we should be able to verify that:
-
StreamRelayowns upstream/local emission, deferred-frame ordering, and sequence numbering, while normalization and item lifecycle state stay in the synchronous pipeline. - upstream response-ID rewriting and tool-name restoration are not applied to already-correct local events; deferred output preserves the gateway-executed tool ordering contract and contiguous sequence numbers.
- client delivery and deferred buffers have explicit entry/byte limits, including handling for a single oversized frame. document retained response-state memory separately rather than claiming queue bounds cap the whole request.
- slow consumers apply backpressure without unbounded spillover or silently dropped semantic events; client disconnect and producer failure release resources and propagate errors without hanging.
- tests exercise deferred/index-less ordering, mixed upstream/local events, tiny queue capacities, oversized frames, slow clients, and cancellation. expose queue occupancy and buffered-byte measurements to tune limits.
the existing client channel is part of this boundary. no spawn_blocking migration is required for this slice.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.