vectordotdev / vectordotdev/vector
vector-to-vector throughput capped by default HTTP/2 flow-control windows
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
A single vector-to-vector (native protocol) connection plateaus at a few MB/s
of event bytes regardless of batch.* and request.concurrency settings.
On our fleet (0.44.0, 2 ms LAN, TLS, end-to-end acknowledgements enabled) each
connection tops out around 7k events/s (~5 MB/s of event bytes) with CPU
nearly idle on both peers.
Evidence that it's flow control rather than processing:
- doubling
batch.max_bytesexactly doubles per-request latency, throughput unchanged request.concurrency1 vs 8 vs 32 makes no difference over the network- N parallel connections scale linearly (~20 connections reach ~45k events/s aggregate)
- on localhost the same binaries and configs sustain 65-74k events/s per connection
The sink builds its hyper client with only .http2_only(true) and the shared
gRPC server builder doesn't set window options either, so both peers run with
the RFC 7540 default 65,535-byte stream/connection windows. With
acknowledgements enabled the server only completes a request (and the client
only gets WINDOW_UPDATE credit back) after events are accepted downstream, so
effective throughput ≈ 64 KiB / (network RTT + ack latency).
Configuration
Version
0.56.0
Debug Output
Example Data
No response
Additional Context
Proposed fix
Enable adaptive (BDP-sized) HTTP/2 windows on the vector sink client and the
shared gRPC server builder — #25782.
References
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the vector sink's hyper client construction and the shared gRPC server builder described in the issue; read how HTTP/2 windows are configured. Compare the current defaults with the proposed adaptive BDP-sized windows in #25782. Done means a single native vector connection no longer plateaus at the reported rate with acknowledgements enabled, with the throughput behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100