connectrpc / connectrpc/connect-rust
server: trailing metadata is echoed unfiltered, and the reserved-name policy needs deciding
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 509
- Forks
- 66
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 10
Description
Summary
#262 stops the server echoing an upstream error's response headers onto the wire, because an echoed content-length made hyper write zero bytes and drop a Connect/HTTP-1.1 connection. The same class exists one frame later, in trailers, and it is not fixed there: err.trailers() and the handler's context trailing metadata are both echoed with no filtering at all.
Measured against real clients — a gateway propagating an upstream failure puts content-length: 4096, connection: keep-alive, transfer-encoding: chunked and a second content-type into the HTTP/2 trailers frame, and they arrive verbatim at both grpc-go v1.81.1 and connect-go v1.19.1. On the Connect unary path the same names arrive as trailer--prefixed headers. Both Go clients tolerate it today, so this is latent rather than breaking, unlike the header case.
An attempt to fix it inside #262 was reverted, because it turned out to need a policy decision rather than a predicate. The attempt is worth reading before redoing it — what follows is what it ran into.
Why this is not just "call the same predicate on trailers"
Provenance is not derivable where the filtering happens. ConnectError::with_trailers is a plain public handler API — "Add response trailers to be included in the error response" — and this repo's own conformance server calls it at six sites. So err.trailers() may hold trailing metadata a handler authored deliberately, or metadata propagated from an upstream, and nothing at the call site distinguishes them. A flag threaded from the call sites tracks which struct field the trailers arrived in, not who wrote them; the attempt did exactly that and the flag was wrong for every error path.
That matters because the two cases want different handling. Silently dropping an upstream's content-length is right. Silently dropping a name a handler deliberately set is a bad afternoon for that handler, with no signal.
But the obvious signal is remotely drivable. Logging a warning when a reserved name is dropped from handler-set metadata sounds right until you notice that handler-set trailer names are frequently derived from request data — the conformance server sets resp.trailers verbatim from the request payload, and real gateways that echo caller-supplied metadata have the same shape. That is one log line per RPC at whatever rate a peer can sustain, unbounded and undeduplicated.
And the source-selection branch has the same bug one level up. build_grpc_trailers decides between the error's trailers and the context's by testing whether the error's are non-empty. Filter after that test and an upstream error whose trailers are entirely reserved names suppresses the handler's own context trailers and emits nothing at all.
What a fix has to decide
- Which names. The header-side list in #262 is hop-by-hop, body framing, provenance (
date), and status-bearing (grpc-status,grpc-message,grpc-status-details-bin). Not all of it transfers: inadd_trailersevery name is rewritten totrailer-{name}before it reaches the wire, so a forwardedcontent-lengthbecomestrailer-content-length, which cannot reach hyper's length encoder or be mistaken for framing by any client. Filtering there is cross-hop consistency, not a fix — worth deciding whether that is a reason to do it or a reason not to. - Whose metadata. Forwarded only, or handler-set too? Forwarded-only is defensible and much smaller. Handler-set filtering is a behaviour change on the success path, which no bug in #202 or #262 reached.
- Observable how. If handler-set names are stripped, a handler needs to find out without giving a peer a log-amplification lever. A
debug!, a once-per-process warning per name, or a compile-time/API-level rejection at the setter are all candidates. The setter is the one place provenance is unambiguous. - What the escape hatch is. If a handler genuinely wants
dateas trailing metadata, is the answer "you cannot", or a namespaced form, or an explicit opt-out?
Notes
server, via and alt-svc were considered and deliberately left forwarded. The argument for leaving them: the deny-list is a correctness boundary where every entry has a mechanical failure attached, and suppressing a few well-known identity headers while forwarding arbitrary upstream metadata does not make a gateway non-leaky — it makes it look non-leaky. A gateway that must not disclose upstream identity should strip metadata wholesale in an interceptor. alt-svc is the one worth revisiting, since a forwarded one redirects the client's future connections.
Conformance says nothing about any of this: the only trailer names anywhere in the upstream testsuite are x-custom-trailer and x-custom-header, so the whole reserved-name question is unexercised. A fix needs its own tests, and the interop harness pattern from #262 (real grpc-go and connect-go clients against our server, with a per-header bisect) is the way to check it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the trailer flow through ConnectError::with_trailers, build_grpc_trailers, and add_trailers, then review the reverted attempt in #262. Resolve the reserved-name, provenance, observability, and escape-hatch policy before changing behavior; add focused trailer tests and use the grpc-go/connect-go interop harness pattern from #262 to verify the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- api, backend-api-design, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100