cloudflare / cloudflare/pingora

Add a per-frame HTTP/2 observation hook to Pingora

Open
#1,010 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
27.4k
Forks
1.7k
Avg merge
6h 22m
Merged PRs (30d)
3

Description

Apologies for the AI-generated request below; I'm not too sure of the exact detail, but the overall request seems sensible to me. Our use-case is true byte-counting per-request (including headers) and this feels like a generic enough solution that may be useful to others also...

## Summary

Expose a callback invoked for every HTTP/2 frame as it crosses the connection,
in both directions, with the frame's raw payload bytes as Pingora saw them and
their decoded detail — a HEADERS block's compressed length and flags, SETTINGS
entries, GOAWAY/RST_STREAM error codes, PING payloads, WINDOW_UPDATE increments,
and so on.

This is a **Pingora-only change**: it is implemented by observing the bytes on
the stream handed to `h2::server::handshake`, so no `h2` release is required.
The one thing that route cannot provide is the decoded header *fields*; see
"Raw bytes and decoded headers" below.

## Why

Pingora gives applications `Session::body_bytes_read()/body_bytes_sent()`
(decoded body bytes) and connection-level `Digest`/socket counters (whole TCP
connection). Neither can describe an individual HTTP/2 stream's wire cost, and
nothing exposes framing at all.

Use cases:

- **Per-stream byte accounting.** HTTP/2 multiplexes streams onto one
connection, so socket counters cannot be attributed per stream. Billing,
quotas, and per-request reporting need each stream's header-block and frame
bytes.
- **HPACK observability and diagnostics.** Compression ratio per stream,
dynamic-table behaviour, and detecting a client that has stopped compressing
are all invisible today.
- **Security research and hardening.** Header-list-size abuse
(`SETTINGS_MAX_HEADER_LIST_SIZE`), HPACK/BREACH-style compression
side-channels, frame-level protocol anomalies, and smuggling or reset abuse
are frame properties. A callback gives defenders the same view an attacker
has.
- **Tracing and troubleshooting.** Per-frame timing (inter-frame gaps, drip
feeds, stalls), stream lifecycle, and why a connection was reset.
- **Adaptive policy.** Priority hints, per-stream rate limiting, and traffic
shaping driven by what actually went on the wire.
- **Fuzzing and conformance.** A generic frame stream is a test oracle for any
HTTP/2 stack.

## Current state

Nothing like this exists in 0.9.0. In
`pingora-core/src/protocols/http/v2/server.rs` there are no occurrences of
`hpack`, `header_block`, `encoded`, or `compressed`; the only header-size symbol
is the decoded `max_header_list_size` receive limit. `handshake` is fixed to
`Stream`, decoded request extensions are dropped, `Digest` is a fixed struct,
and `HttpPersistentSettings::user_context` is HTTP/1-only.

## Proposed interface

A single observer with a default no-op method, so an implementation overrides
only the frames it handles:

```rust
pub enum Direction { Read, Write }

/// Everything known about one frame. `raw` is the frame's payload exactly as
/// Pingora observed it on the wire — padding included, the 9-byte frame header
/// excluded — so an observer can inspect the bytes directly. `kind` is the
/// decoded view of the same frame.
pub struct FrameEvent<'a> {
pub direction: Direction,
pub stream_id: u32,
pub at: Instant,
pub raw: &'a [u8],
pub kind: FrameKind<'a>,
}

pub enum FrameKind<'a> {
Headers {
/// Compressed HEADERS + CONTINUATION octets, padding/priority excluded.
header_block_len: u32,
end_headers: bool,
end_stream: bool,
},
Data { end_stream: bool },
Settings { ack: bool, entries: &'a [Setting] },
WindowUpdate { increment: u32 },
Priority { dependency: u32, weight: u8, exclusive: bool },
RstStream { error_code: u32 },
Ping { ack: bool, opaque: [u8; 8] },
GoAway { last_stream_id: u32, error_code: u32, debug: &'a [u8] },
PushPromise { promised_id: u32, header_block_len: u32 },
Unknown { frame_type: u8 },
}

pub trait FrameObserver: Send + Sync + 'static {
fn on_frame(&self, _event: FrameEvent<'_>) {}
}
```

Ergonomic variant: one default method per frame type (`on_headers`, `on_data`,
`on_settings`, ...) instead of a single `on_frame`, backed by the same enum.

Registration is opt-in and defaults off, e.g. through Pingora's HTTP/2 options:

```rust
H2Options::default()
.max_header_list_size(64 * 1024)
.frame_observer(Arc::new(MyObserver::new()));
```

## Where it lives

Pingora already owns the `Stream` it passes to `h2::server::handshake`
(`apps/mod.rs`), and the same seam exists for the upstream client
(`connectors/http/v2.rs`). A transparent `AsyncRead`/`AsyncWrite` wrapper around
that stream observes frames without touching `h2`:

- It sits above TLS, so it sees cleartext HTTP/2 framing for both h2c and TLS.
- It parses the 9-byte frame header and reassembles each frame's payload so it
can hand the raw bytes to the observer. One frame is buffered per direction,
bounded by the connection's `SETTINGS_MAX_FRAME_SIZE`, and the buffer is
reused; capture can be restricted to selected frame types (for example,
control frames and HEADERS) so a bulk DATA path does not pay for a copy.
- Non-HPACK frames are trivially decoded from their payloads (SETTINGS,
WINDOW_UPDATE, PRIORITY, RST_STREAM, PING, GOAWAY); HEADERS/CONTINUATION
contribute their accumulated payload length and flags, and their raw
compressed bytes.

Details the implementation must handle:

- **Partial writes** — parse only the accepted prefix of `buf`; never account
for bytes the inner IO did not take.
- **Vectored writes** — handle `poll_write_vectored` without double-counting.
- **Partial reads** — the parser is a byte-stream state machine, so chunk
boundaries do not matter.
- **Client preface** — skip the 24-byte `PRI * HTTP/2.0...` preface (h2c
`try_peek` does not consume it).
- **Trailers** — a later HEADERS on a known stream is a trailer block; report it
as its own event.
- **`PUSH_PROMISE`** — include it, or skip by length so framing stays in sync.

## Raw bytes and decoded headers

`FrameEvent::raw` carries each frame's payload exactly as observed, so HEADERS
and CONTINUATION events expose the raw compressed block bytes and every other
frame type exposes its raw payload. What the wrapper cannot produce is the
*decoded* header fields, because HPACK dynamic-table state lives inside `h2`;
the decoded request/response headers remain available as they are today via
`Session::req_header()` / `response_written()`.

A small `h2` change could go further — store the fragment length in
`Headers::load` and surface the decoded `HeaderMap` on the callback, at the
point the frame is processed. That is the clean way to expose decoded fields
per frame, but it is **not needed for the accounting, observability, and
security use cases above**, so it can be left as future work.

## Compatibility and cost

- Purely additive and opt-in; no behaviour change and no cost when disabled.
- One callback per frame on the connection task; observers must be cheap and
non-blocking (batch into atomics/counters if needed).
- Reassembling raw payloads copies at most one frame per direction into a
reused buffer, bounded by `SETTINGS_MAX_FRAME_SIZE` (16 KiB by default;
cap it rather than trusting a peer's advertised value). Capture filtering
keeps bulk DATA off that path when it is not needed.

Contributor guide

Open the contributing guide

Research direction

Start by reading pingora-core/src/protocols/http/v2/server.rs and the stream handoff in apps/mod.rs, then compare the upstream seam in connectors/http/v2.rs. Confirm how an opt-in observer can wrap the streams in both directions while preserving partial reads, partial and vectored writes, the client preface, continuation frames, trailers, and PUSH_PROMISE framing; done means the proposed raw and decoded frame events work without cost when disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, networking, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.