cloudflare / cloudflare/pingora

Hold final response headers until an async body verdict is ready

Open
#978 0 comments 1 reaction 1 assignee Claimed by @andrewhavck View on GitHub
enhancement
Dominant language
Rust
Stars
27.4k
Forks
1.7k
Avg merge
6h 22m
Merged PRs (30d)
3

Description

## What is the problem your feature solves, or the need it fulfills?

Making `upstream_response_body_filter` async (#964 / #974) allows a proxy to await response-body inspection, but the upstream status and headers may already be written downstream by the time the verdict is ready. A response that began as 200, for example, can no longer become a 403.

The response pipeline could be extended with an opt-in way to hold the final upstream header. Care would need to be taken to avoid breaking informational responses, protocol framing, downstream modules, cache behavior, or bounded streaming of large bodies.

## Describe the solution you'd like

Add a request-scoped policy to `ProxyHttp`:

```rust
fn response_commit_policy(
&self,
session: &Session,
ctx: &Self::CTX,
) -> ResponseCommitPolicy {
ResponseCommitPolicy::Immediate
}
```

I have a draft implementation available on [`deferred-response-header`](https://github.com/torinnd/pingora/tree/deferred-response-header), stacked on #974. I'd be happy to open this as a PR if the direction looks suitable.

In that draft, `ResponseCommitPolicy::Hold` bypasses cache lookup and admission so the response reaches the upstream inspection hooks. Cache management operations such as PURGE keep their normal behavior.

After `response_filter` and protocol framing, Pingora holds the transformed final header. Dedicated body and trailer hooks receive capability types instead of `&mut Session`:

- `HeldResponseBody` owns the current body task and exposes read-only session state plus the held header, but no writer.
- `commit()` consumes it and leaves the body task for Pingora.
- `commit_for_streaming()` clears the body task and returns `CommittedResponse`.
- `CommittedResponse::write_chunk()` writes non-terminal chunks; Pingora supplies EOS.

Pingora flushes earlier informational responses before invoking a held hook. Header-only and 101 responses pass through immediately. Missing verdicts, premature writes, invalid informational mutations, and conflicting framing fail closed. The same gate transitions are used for h1, h2, and custom upstream responses.

## Describe alternatives you've considered

I'll call out a few lessons-learned / false starts I went down when developing that draft:

### Capabilities rather than writer checks

A gate represented only by `Session` state requires every response-writing method to check that state. Capability types make the transition explicit: a held hook has no writer, and write access appears only when `HeldResponseBody` is consumed.

### Capture after header transformation

Capturing before `response_filter` loses application changes and protocol framing; capturing in the downstream writer is too late when a header and body share a filtered batch. The proposed gate captures after transformation but before body filtering.

### Commit through the ordered response path

Writing directly from the body hook can overtake an informational response still queued in the same batch. The gate flushes earlier informational responses before invoking the held hook, then commits the final header through downstream modules.

### Mutate the held header rather than replace it

An arbitrary replacement header bypasses framing and `response_filter` work already applied to the upstream response. The proposed API exposes the transformed header for mutation and validates basic framing conflicts at commit.

### Bypass cache rather than inspect cache hits

Cache hits use the downstream body-filter path rather than `upstream_response_body_filter`. Async cache-hit inspection would require a separate cache-serving change, so held requests bypass lookup and admission.

## Additional context

`ResponseCommitPolicy::Immediate` remaining the default should result in very little disruption for those who don't need this functionality. This can be implemented with additional state branches but without any allocations.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.