cloudflare / cloudflare/cloudflared
🐛 QUIC request body `Close` does not unblock a pending `Read`, causing streaming requests to hang with `http2Origin`
- Dominant language
- Go
- Stars
- 15.6k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
## Describe the bug
When the edge-to-cloudflared transport is QUIC and cloudflared connects to an origin using HTTP/2 (`http2Origin: true`), a streaming request can receive the complete origin response payload but never receive response EOF.
The trigger is an HTTP/2 origin that finishes its response while the request body remains open and idle. No reverse proxy, gRPC server, or application framework is required.
On the QUIC path, cloudflared presents `nopCloserReadWriter` as the origin request body. Its `Close` method only sets an atomic flag:
```go
func (np *nopCloserReadWriter) Close() error {
atomic.StoreUint32(&np.closed, 1)
return nil
}
```
This prevents a future `Read`, but it does not interrupt a `Read` already blocked in the underlying QUIC stream. This violates the `net/http` request-body contract, which requires `Close` to unblock a concurrent `Read`.
With an HTTP/2 origin, response cleanup can wait for the request-body writer to exit. The real QUIC `CancelRead(0)` occurs only during the outer stream cleanup, which cannot run until the proxy handler returns:
```text
HTTP/2 origin response cleanup
-> waits for the request-body writer
-> writer is blocked in the incoming QUIC stream Read
-> CancelRead occurs only in outer runStream cleanup
-> runStream cleanup waits for the proxy handler to return
```
## To Reproduce
### End-to-end reproducer using a raw HTTP/2 origin
[cloudflared-quic-h2-minimal-repro.zip](https://github.com/user-attachments/files/29929287/cloudflared-quic-h2-minimal-repro.zip)
The reproducer consists of two small Go programs and has no external Go dependencies:
- A raw TLS HTTP/2 origin.
- An HTTP/2 streaming client.
The raw origin:
1. Negotiates `h2` with ALPN.
2. Waits until it receives the first non-empty request DATA frame.
3. Waits briefly so cloudflared's origin request writer blocks on its next request-body read.
4. Sends `:status: 200` and the payload `complete-body\n`.
5. Sends response `END_STREAM` without resetting or closing the request direction.
6. Continues accepting request DATA.
The client declares a 1 MiB request body, sends one byte, and deliberately leaves the remainder open. It verifies separately that the response payload and response EOF arrive.
1. Start the raw origin:
```bash
./repro-origin \
-mode h2 \
-addr 127.0.0.1:8443 \
-cert cert.pem \
-key key.pem
```
2. Verify the origin directly. This should pass:
```bash
./repro-client \
-url https://localhost:8443/ \
-insecure \
-eof-timeout 3s
```
Expected:
```text
response payload: "complete-body\n"
PASS: response EOF arrived promptly
```
3. Configure a dedicated test Tunnel hostname to connect directly to the raw origin:
```yaml
tunnel:
credentials-file: /path/to/.json
ingress:
- hostname:
service: https://127.0.0.1:8443
originRequest:
http2Origin: true
noTLSVerify: true
- service: http_status:404
```
4. Start cloudflared with the edge transport forced to QUIC:
```bash
cloudflared tunnel \
--protocol quic \
--metrics 127.0.0.1:2000 \
--config ./config.yml \
run
```
5. Run the streaming client through the Tunnel:
```bash
./repro-client \
-url https:/// \
-eof-timeout 3s \
-total-timeout 20s
```
6. The suspected failure has this signature:
```text
response payload: "complete-body\n"
REPRODUCED: complete response body arrived, but no EOF after 3s
```
7. Keep `http2Origin: true` unchanged and restart only cloudflared with `--protocol http2`. Repeat the request. Response EOF should now arrive promptly.
8. Tunnel ID: ``
9. cloudflared configuration: shown above with hostname and credential path redacted.
Use a normal dedicated Tunnel hostname. Accountless Quick Tunnels may reject or buffer deliberately unfinished streaming requests before they reach cloudflared, which tests a different edge behavior.
### Deterministic local regression test
The underlying defect can be reproduced without a Cloudflare account or network connection. Add a test that starts a read, waits until it is blocked in the underlying reader, calls `nopCloserReadWriter.Close`, and requires the read to return.
Against cloudflared `2026.7.1`, the test fails consistently:
```text
--- FAIL: TestNopCloserReadWriterCloseUnblocksPendingRead (0.25s)
quic_connection_test.go:621: nopCloserReadWriter.Close did not unblock the pending Read
FAIL
```
The same failure occurs with `go test -race`.
The existing `nopCloserReadWriter` tests only verify that a read started *after* `Close` fails. They do not verify that `Close` wakes a read already in progress.
### Why the nghttp2 command-line tools are not an equivalent reproducer
I also tested whether the issue could be reproduced without the two small Go programs by using the standard nghttp2 command-line client and server. Those tools do not preserve the required stream lifetime:
1. `nghttp --data=-` reads stdin to EOF before opening the HTTP/2 request. Keeping stdin open therefore prevents the request from starting instead of creating an active request with an unfinished body.
2. `curl --upload-file -` can create the required active unfinished HTTP/2 upload, but `nghttpd --early-response` does not preserve the request direction after finishing its response.
3. After its response, `nghttpd` immediately resets the stream:
```text
send HEADERS
send DATA ; END_STREAM
send RST_STREAM error_code=NO_ERROR
```
4. In a public QUIC-edge/HTTP2-origin test, cloudflared received that reset and logged:
```text
error="stream error: stream ID 1; NO_ERROR; received from peer"
```
The reset explicitly cancels the still-open request direction, wakes the request-body writer, and masks the defect. An nghttp2-only four-row matrix therefore tests different teardown semantics and cannot confirm or refute this issue.
The raw origin in the reproduction is intentionally small because it must do something the packaged server does not expose as an option: send response `END_STREAM`, omit `RST_STREAM`, and continue accepting request DATA. The raw client must likewise start the request immediately, send one byte, keep its body open, and independently check whether response EOF arrives.
### Protocol controls
Local integration tests produced this matrix while keeping the early response and unfinished request body unchanged:
| Edge to cloudflared | cloudflared to origin | Result |
|---|---|---|
| QUIC | HTTP/2 | **Pending read is not interrupted** |
| HTTP/2 | HTTP/2 | Completes promptly; 50/50 race-enabled runs passed |
| QUIC | HTTP/1.1 | Completes after approximately 50 ms |
| HTTP/2 | HTTP/1.1 | Completes after approximately 50 ms; 20/20 race-enabled runs passed |
For the HTTP/1.1-origin controls, Go's `net/http.maxWriteWaitBeforeConnReuse` 50 ms timer expires. The origin connection is marked non-reusable and response EOF is released. This avoids the hang but does not fix the QUIC body-close behavior.
## Expected behavior
Closing the origin request body should interrupt any concurrent read from the incoming QUIC request stream while leaving the QUIC response-writing direction available.
When an HTTP/2 origin completes its response before the streaming request upload finishes, cloudflared should forward both the complete response payload and response EOF without waiting indefinitely for another request-body byte.
## Environment and versions
- OS: macOS 26.5.2 (build 25F84)
- Architecture: ARM64
- cloudflared: 2026.7.1, built 2026-07-09
- cloudflared source commit tested: `ecb88678f1368581964ca2bb55615502a2ed2dbe`
- Go: 1.26.5 darwin/arm64
## Logs and errors
No cloudflared error is necessarily logged. The visible symptom is that the complete response payload arrives but response EOF does not.
During the deterministic test, the pending read remains blocked until the mock underlying stream is released. In a live goroutine dump, the expected blocked read is through:
```text
connection.(*nopCloserReadWriter).Read
quic.(*SafeStreamCloser).Read
```
with HTTP/2 origin response cleanup waiting for the request-body writer to finish.
## Additional context
`runStream` deliberately wraps `SafeStreamCloser` in `nopCloserReadWriter` so an inner HTTP request-body close cannot prematurely close the QUIC response-writing half. That separation is necessary, but the receive half still needs an interruptible close operation.
A possible fix is to add a read-side-only close method to `SafeStreamCloser` that calls the underlying QUIC stream's `CancelRead(0)`, and have `nopCloserReadWriter.Close` invoke it exactly once. The existing outer full-stream cleanup should remain responsible for closing the write side.
A regression test should verify that:
1. A read is already blocked in the underlying QUIC stream.
2. Calling `nopCloserReadWriter.Close` wakes that read promptly.
3. The read returns an error.
4. The QUIC write side remains usable so the origin response can still be forwarded.
Contributor guide
Research direction
Start with nopCloserReadWriter and SafeStreamCloser, then run the regression described in quic_connection_test.go, including with -race. Trace how runStream closes the stream and verify that closing the request body wakes a pending read, returns an error, and leaves the response-writing side usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100