vllm-project / vllm-project/agentic-api

refactor: extract StreamRelay and bound client delivery

Open
#244 4 comments 0 reactions 1 assignee View on GitHub

@LOGO127 is already working on this.

Since Sep 14, 2026.

Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

ingestion and client emission currently share control flow, while the final client-facing channel is unbounded. bounding proposed worker queues alone would still allow slow-client buffering to grow. we should isolate emission and make backpressure reach the client boundary, independently of any worker-placement decision.

proposed second slice of #241, building on the synchronous ingestion boundary in #243. this is a suggested scope; feel free to revise, combine, or remove it.

we should be able to verify that:

  • StreamRelay owns upstream/local emission, deferred-frame ordering, and sequence numbering, while normalization and item lifecycle state stay in the synchronous pipeline.
  • upstream response-ID rewriting and tool-name restoration are not applied to already-correct local events; deferred output preserves the gateway-executed tool ordering contract and contiguous sequence numbers.
  • client delivery and deferred buffers have explicit entry/byte limits, including handling for a single oversized frame. document retained response-state memory separately rather than claiming queue bounds cap the whole request.
  • slow consumers apply backpressure without unbounded spillover or silently dropped semantic events; client disconnect and producer failure release resources and propagate errors without hanging.
  • tests exercise deferred/index-less ordering, mixed upstream/local events, tiny queue capacities, oversized frames, slow clients, and cancellation. expose queue occupancy and buffered-byte measurements to tune limits.

the existing client channel is part of this boundary. no spawn_blocking migration is required for this slice.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.