vectordotdev / vectordotdev/vector

vector-to-vector throughput capped by default HTTP/2 flow-control windows

Open
#25,792 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

A note for the community
  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem

A single vector-to-vector (native protocol) connection plateaus at a few MB/s
of event bytes regardless of batch.* and request.concurrency settings.
On our fleet (0.44.0, 2 ms LAN, TLS, end-to-end acknowledgements enabled) each
connection tops out around 7k events/s (~5 MB/s of event bytes) with CPU
nearly idle on both peers.

Evidence that it's flow control rather than processing:

  • doubling batch.max_bytes exactly doubles per-request latency, throughput unchanged
  • request.concurrency 1 vs 8 vs 32 makes no difference over the network
  • N parallel connections scale linearly (~20 connections reach ~45k events/s aggregate)
  • on localhost the same binaries and configs sustain 65-74k events/s per connection

The sink builds its hyper client with only .http2_only(true) and the shared
gRPC server builder doesn't set window options either, so both peers run with
the RFC 7540 default 65,535-byte stream/connection windows. With
acknowledgements enabled the server only completes a request (and the client
only gets WINDOW_UPDATE credit back) after events are accepted downstream, so
effective throughput ≈ 64 KiB / (network RTT + ack latency).

Configuration

Version

0.56.0

Debug Output

Example Data

No response

Additional Context

Proposed fix

Enable adaptive (BDP-sized) HTTP/2 windows on the vector sink client and the
shared gRPC server builder — #25782.

References

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the vector sink's hyper client construction and the shared gRPC server builder described in the issue; read how HTTP/2 windows are configured. Compare the current defaults with the proposed adaptive BDP-sized windows in #25782. Done means a single native vector connection no longer plateaus at the reported rate with acknowledgements enabled, with the throughput behavior verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.