hyperium / hyperium/h2

Sending and receiving data frames introduces significant overhead

Open
#902 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.6k
Forks
382
Avg merge
20h 23m
Merged PRs (30d)
9

Description

What is the issue?

I was looking at the perf of requests & responses sent via h2 on a TCP stream versus the perf of raw messages sent back and forth over a TCP stream,
and found throughput with h2 to be reduced by an average of around 50%.
The following table compares the perf that I observed on the test VM that I used with h2 requests & responses versus raw messages:

OS: linux (kernel version 6.6)
Arch: arm64
TCP_NODELAY: True
Data Payload Size | Message Kind | Avg Throughput | Avg Round-Trip Latency
   4kiB           | h2 over TCP  | 0.97 GB/s      |   8.43 µs
   4kiB           | raw over TCP | 1.84 GB/s      |   4.45 µs
  16kiB           | h2...        | 2.68 GB/s      |  12.23 µs
    ...           | raw...       | 5.31 GB/s      |   6.17 µs
  64kiB           | h2...        | 3.54 GB/s      |  37.05 µs
    ...           | raw...       | 8.43 GB/s      |  15.54 µs
 256kiB           | h2...        | 4.64 GB/s      | 113.00 µs
    ...           | raw...       | 9.43 GB/s      |  55.57 µs
1024kiB           | h2...        | 4.28 GB/s      | 490.00 µs
    ...           | raw...       | 9.22 GB/s      | 227.40 µs

Primary root causes

Currently, the encoder can only hold a single pending DATA frame, and blocks the acceptance of any new frame until the
previous one is fully flushed to the socket and reclaimed.
This means every DATA frame triggers its own write() syscall and TCP segment — even when the TCP send buffer could carry far more.
With TCP_NODELAY enabled, there is no Nagle buffering to coalesce these small writes, so the per-frame overhead is fully realized.

On the receiver side, the recv buffer used by the frame decoder is small by default, and even if its buffer size was increased,
the effective capacity of that buffer would degrade over time and never recover, causing small read() calls.
This is because:

  1. The buffer is repeatedly split to yield decoded frames.
  2. If the recv buffer runs out of capacity, BytesMut::reserve is called, & if that buffer is still in use (likely),
    the current buffer is abandoned & a new buffer with a maximum capacity of 64kB would be created
    due to how BytesMut encodes its original capacity to a max of 64kB & caps new reservations
    based on that encoded original capacity.

Together, these things inflate the syscall-to-payload ratio on both ends.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported h2-versus-raw TCP measurements first, then inspect the encoder's single pending DATA frame behavior and the frame decoder's recv buffer and BytesMut::reserve usage. Done means reducing the syscall-to-payload overhead on both sending and receiving sides and validating improved throughput and round-trip latency against the reported comparisons.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.