vllm-project / vllm-project/agentic-api
perf: benchmark inline versus worker placement for streaming ingestion
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 284
- Forks
- 74
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 93
Description
the current server-sent events (SSE) benchmark exercises the offloaded accumulator under concurrency, but it has no equivalent inline SSE control and uses an in-memory stream. that doesn't establish whether moving the live pipeline to a worker improves the workloads we care about. we should compare placement after the synchronous pipeline exists, then record whether offloading is worth implementing.
proposed third slice of #241, using the synchronous core from #243. relay extraction and client backpressure do not depend on a positive result here. this is a suggested scope; feel free to revise, combine, or remove it.
we should be able to verify that:
- inline and worker variants run equivalent normalize/validate/fold/translate work with the same inputs and correctness checks; include both isolated processing and end-to-end measurements.
- workloads cover realistic event sizes and arrival pacing, concurrent streams, long outputs, and slow consumers. report throughput, tail event latency, CPU use, memory, and worker/thread counts with reproducible commands and environment details.
- experiments sweep bounded queue capacities with byte budgets and occupancy measurements; 16 entries is a candidate to test, not a prescribed default.
- the experimental worker driver feeds input and drains output concurrently, passes tiny-capacity deadlock tests, and has explicit disconnect, cancellation, shutdown, and join/error handling. don't rely on aborting an already-running
spawn_blockingtask. - a recorded placement recommendation explains the tradeoffs against agreed latency/resource goals. keeping the live path inline is a valid outcome; production offloading is a separate follow-up only if the evidence supports it.
the existing benchmark is the baseline to extend.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with crates/agentic-server-core/benches/accumulator_throughput.rs and the synchronous core from #243. Extend the benchmark to compare equivalent inline and worker processing across the listed workloads, queue capacities, and cancellation cases, using reproducible commands and environment details. Done means the measurements and recommendation are recorded, whether or not production offloading is advised.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100