vllm-project / vllm-project/agentic-api

perf: benchmark inline versus worker placement for streaming ingestion

Open
#245 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

the current server-sent events (SSE) benchmark exercises the offloaded accumulator under concurrency, but it has no equivalent inline SSE control and uses an in-memory stream. that doesn't establish whether moving the live pipeline to a worker improves the workloads we care about. we should compare placement after the synchronous pipeline exists, then record whether offloading is worth implementing.

proposed third slice of #241, using the synchronous core from #243. relay extraction and client backpressure do not depend on a positive result here. this is a suggested scope; feel free to revise, combine, or remove it.

we should be able to verify that:

  • inline and worker variants run equivalent normalize/validate/fold/translate work with the same inputs and correctness checks; include both isolated processing and end-to-end measurements.
  • workloads cover realistic event sizes and arrival pacing, concurrent streams, long outputs, and slow consumers. report throughput, tail event latency, CPU use, memory, and worker/thread counts with reproducible commands and environment details.
  • experiments sweep bounded queue capacities with byte budgets and occupancy measurements; 16 entries is a candidate to test, not a prescribed default.
  • the experimental worker driver feeds input and drains output concurrently, passes tiny-capacity deadlock tests, and has explicit disconnect, cancellation, shutdown, and join/error handling. don't rely on aborting an already-running spawn_blocking task.
  • a recorded placement recommendation explains the tradeoffs against agreed latency/resource goals. keeping the live path inline is a valid outcome; production offloading is a separate follow-up only if the evidence supports it.

the existing benchmark is the baseline to extend.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/agentic-server-core/benches/accumulator_throughput.rs and the synchronous core from #243. Extend the benchmark to compare equivalent inline and worker processing across the listed workloads, queue capacities, and cancellation cases, using reproducible commands and environment details. Done means the measurements and recommendation are recorded, whether or not production offloading is advised.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.