Streamed PUTs intermittently 520: Cloudflare mints `error code: 520` on the worker's egress leg

Symptom

~0.1% of requests fail, exclusively PutObject / UploadPart. Clients see 520 (majority) or 503, with the upload aborted mid-flight. Present since at least 2.3.2 and unchanged by the multistore 0.7.2 bump — that release fixed federated reads; multistore-cf-workers, which owns the write path, was byte-identical between 0.7.0 and 0.7.2.

Affects both request shapes and both operations, from two unrelated clients/SDKs:

• plain SigV4 PUT with Content-Length (→ FixedLengthStream branch)
• aws-chunked + STREAMING-UNSIGNED-PAYLOAD-TRAILER (→ raw set_body(stream) branch)

Body sizes 593 KB – 64 MiB, so it is "has a streamed body", not "is large".

What the logging established

The 5xx WARN added in #201/#204 records the relayed backend response. The gateway passes the upstream status through verbatim (GatewayResponse::Forward → response_from_forward), and handle_request returns normally — the worker is not being killed. Two independent invocations, identical:

status=520  duration_ms=79    content_length="16777216"   (UploadPart)
status=520  duration_ms=138   content_length="4941808"    (PutObject)

resp_server=cloudflare
resp_cf_ray=<a different ray than inbound>-CDG
resp_amz_request_id=      (empty)
resp_amz_id_2=            (empty)
resp_content_type=text/plain; charset=UTF-8
resp_content_length=16          # "error code: 520\n"

Therefore:

• Cloudflare mints the 520 on the worker's own egress leg. Not the inbound edge, not S3.
• S3 never produces an HTTP response. Zero x-amz-* on any sample.
• The 16-byte text/plain body is Cloudflare's bare error page — the same artifact the STS_REQUEST_TIMEOUT comment in src/lib.rs already describes, and the reason callers' AWS SDKs fail to deserialize instead of retrying.
• It dies in 79–138 ms, roughly 380 KB into a multi-MB body, so the outbound request collapses just after headers.

Cloudflare's docs define 520 as the origin returning an empty, unknown, or unexpected response — including one "lacking an HTTP status code or response body".

Ruled out

• S3 returning 520 or any error (it never answers).
• The inbound edge killing the worker (handle_request returns normally).
• A broken uploader (two unrelated clients, different SDKs and signing modes).
• Cross-request I/O-context contamination as the cause. It is real and reproducible (tests/repro/streamed-put-io-context/, #203) but produces Network connection lost → BackendError → 503, the minority mode (2 of 23), not the 520s.

Open question

Why the Cloudflare↔S3 leg yields an empty response.

The one uninstrumented spot on our side is the dropped promise in WorkerBackend::forward (multistore-cf-workers):

let _ = stream.pipe_to(&transform.writable());

If that pipe fails early, nothing captures why — and a truncated outbound body is the kind of thing that makes an origin hang up without responding. Capturing that rejection is the next diagnostic, and it is upstream.

Failing that, the subrequest ray IDs (a22766ebec7ee989-CDG, a22775716c8322a5-CDG) are what Cloudflare support needs to trace the egress leg. They exist only because of #204.

Secondary

TypeError: Can't read from request stream after responding with an exception fires ~1 ms after the response commits — the orphaned pipe still reading the inbound body. A consequence rather than the cause, but it is what resets the connection so the client sees a broken pipe instead of the relayed status.

Start with WorkerBackend::forward in multistore-cf-workers, especially the dropped pipe_to promise, and review src/lib.rs for the STS_REQUEST_TIMEOUT context. Run tests/repro/streamed-put-io-context/ and compare its 503 behavior with the 520 traces and subrequest ray IDs from #204. Done means the pipe rejection is captured and its relationship to the empty Cloudflare response is established.