cloudflare / cloudflare/pingora

Upstream timeouts (connect/read/write) synthesize 502 instead of 504 in fail_to_proxy's default status mapping

Open
#980 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
27.4k
Forks
1.7k
Avg merge
6h 22m
Merged PRs (30d)
3

Description

## Describe the bug

The default `fail_to_proxy` implementation maps every upstream-sourced error to 502:

https://github.com/cloudflare/pingora/blob/pingora-proxy-0.8.1/pingora-proxy/src/proxy_trait.rs#L489-L516

```rust
ErrorSource::Upstream => 502,
```

This means `ConnectTimedout`, `TLSHandshakeTimedout`, `ReadTimedout`, and `WriteTimedout` — all deadline expiries against the upstream — surface to clients as 502 Bad Gateway.

RFC 9110 §15.6.6 defines 504 specifically for this case: *"the server, while acting as a gateway or proxy, did not receive a timely response from an upstream server."* A timed-out upstream and a broken upstream are different failure modes, and the status code is the only signal most clients and dashboards get.

For reference, both nginx and Envoy answer 504 when their upstream connect/read/write deadlines expire (nginx: `proxy_connect_timeout` / `proxy_read_timeout` / `proxy_send_timeout`; Envoy: route/cluster timeouts map to gateway-timeout responses).

## Pingora info

**Pingora version**: 0.8.1
**Rust version**: any (mapping is version-independent since the hook exists)
**Operating system version**: any

## Steps to reproduce

1. Proxy a request to an upstream that accepts the TCP connection but never answers.
2. Let the read timeout expire.

Minimal reproduction with the stock `ProxyHttp` implementation (no `fail_to_proxy` override):

```console
$ curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/ # upstream accepts, never responds
502
```

With the default `read_timeout` of 30s the client waits out the deadline and then receives:

```
HTTP/1.1 502 Bad Gateway
```

## Expected results

Deadline expiries against the upstream answer **504 Gateway Timeout**:

```rust
ErrorSource::Upstream => match e.etype() {
ErrorType::ConnectTimedout
| ErrorType::TLSHandshakeTimedout
| ErrorType::ReadTimedout
| ErrorType::WriteTimedout => 504,
_ => 502,
},
```

Everything else upstream-sourced (refused, unroutable, TLS failures, mid-body resets) keeps 502.

## Observed results

502 for both classes, so:

- monitoring cannot distinguish "upstream slow/hung" from "upstream broken";
- browsers render a timeout as "Bad gateway", which misleads users;
- operators building on `ProxyHttp` who want the RFC-compliant mapping have to copy the entire default mapping into an override just to add the timeout arms (and keep the copy in sync across upgrades).

A secondary suggestion attached to the same code: extracting the default mapping into a public helper (e.g. `pub fn default_failure_status(e: &Error) -> u16`) would let implementations reuse the mapping and only override the parts they disagree with, instead of forking the whole hook body.

Happy to send a PR for the mapping change, the helper extraction, or both, if the direction sounds right.

Contributor guide

Open the contributing guide

Research direction

Start at pingora-proxy/src/proxy_trait.rs#L489-L516 and inspect the default fail_to_proxy status mapping. Exercise the timeout and non-timeout upstream errors described in the issue, using the existing test entry points if available. Done means upstream deadline expiries produce 504 while other upstream-sourced failures remain 502; decide separately whether the public helper belongs in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.