Small unary RPC fixed cost versus raw h2 gRPC-like path
- Dominant language
- Rust
- Stars
- 12.5k
- Forks
- 1.3k
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 24
Description
# Small unary RPC fixed cost: tonic unary vs raw h2 gRPC-like loopback path
## Summary
While investigating high-rate 1KiB metadata-style internode RPCs in RustFS, we isolated a small unary RPC benchmark to separate application wrapper cost from tonic/h2 fixed cost.
The repro compares three loopback lanes:
- `tonic_unary`: generated tonic unary client/server over HTTP/2.
- `tonic_unary_boxed_tower_wrapper`: the same tonic unary path with a minimal boxed tower service wrapper around the client transport.
- `raw_h2_grpc_like_prefix_headers_trailers`: direct `h2` client/server with gRPC-like request/response headers, a 5-byte gRPC message prefix, prost encode/decode, and `grpc-status: 0` response trailers.
The raw h2 lane is not production-equivalent to tonic. It is intended as a narrow fixed-cost probe for high-rate small unary calls.
## Versions Tested
- tonic `0.14.6`
- h2 `0.4.18`
- prost `0.14.4`
- tokio `1.53.1`
- payload: 1024 bytes
- measured calls per lane: 20,000 after 1,000 warmup calls
- environment: macOS arm64 local loopback
## Results
Three consecutive local release runs:
```text
tonic_unary: calls=20000 payload=1024B avg=99799ns p50=97750ns p90=109334ns p99=122417ns
tonic_unary_boxed_tower_wrapper: calls=20000 payload=1024B avg=99406ns p50=97458ns p90=108917ns p99=121333ns
raw_h2_grpc_like_prefix_headers_trailers: calls=20000 payload=1024B avg=74503ns p50=73708ns p90=85792ns p99=99416ns
tonic_unary: calls=20000 payload=1024B avg=101505ns p50=99458ns p90=112042ns p99=125542ns
tonic_unary_boxed_tower_wrapper: calls=20000 payload=1024B avg=99951ns p50=98125ns p90=109292ns p99=121667ns
raw_h2_grpc_like_prefix_headers_trailers: calls=20000 payload=1024B avg=73356ns p50=71250ns p90=84583ns p99=96416ns
tonic_unary: calls=20000 payload=1024B avg=101963ns p50=98709ns p90=111958ns p99=152166ns
tonic_unary_boxed_tower_wrapper: calls=20000 payload=1024B avg=100227ns p50=97875ns p90=109875ns p99=124209ns
raw_h2_grpc_like_prefix_headers_trailers: calls=20000 payload=1024B avg=74070ns p50=72125ns p90=85709ns p99=99417ns
```
In this repro, the minimal boxed tower wrapper does not add measurable overhead. The repeatable gap is about 25-28 microseconds per 1KiB unary call between generated tonic unary and direct h2 with gRPC-like framing/trailers plus prost encode/decode.
## Question
Is there a known lighter unary path, configuration, or avoidable allocation/readiness/body abstraction cost in tonic `Grpc::unary` for high-rate small unary RPCs?
If the gap is mostly required by full gRPC/tonic semantics, that would also be useful confirmation. For RustFS, it would push the optimization direction toward reducing per-object RPC count at the application fanout layer rather than tuning h2 windows/pooling or micro-optimizing caller-side fixed headers.
## Reproducer
```bash
cargo run --release
```
Contributor guide
Research direction
Run the reproducer with cargo run --release and compare the tonic_unary, boxed tower wrapper, and raw_h2_grpc_like_prefix_headers_trailers lanes described in the issue. Read the tonic Grpc::unary path and its surrounding transport abstractions first; done means identifying a lighter supported path or configuration, or confirming that the gap is required by tonic and gRPC semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- backend-api-design, networking, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100