cloudflare / cloudflare/cloudflared

🐛 GOAWAY frames not handled correctly with http2

Open
#1,273 0 comments 0 reactions 0 assignees View on GitHub
Priority: Normal Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
When connecting to nginx with https using http2 the following error appears in the logs intermittently and results in client facing 502 errors;

`Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error`

**To Reproduce**
Steps to reproduce the behavior:
1. Install nginx container (such as nginx:alpine) and cloudflared:latest
2. Use an nginx config file with http2 on (Using a short timeout or request limit triggers GOAWAY frames faster);
```
server {
listen 443 ssl default_server;
http2 on;
keepalive_timeout 1s;
keepalive_requests 1;
}
```
3. Use a cloudflared config file with http2 on;
```
ingress:
- service: https://nginx:443
originRequest:
http2Origin: true
```
4. Start sending requests through cloudflared and view logs. You will see the error;
```
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
```

5. Setting `http2Origin: false` resolves the issue.

**Expected behavior**
Cloudflared should see the GOAWAY frame and retry the request.

**Environment and versions**
- Cloudflared Version: 2024.6.0

**Additional context**
Appears to be an issue with `http.Request` missing a no-op `GetBody()` method.

The GOAWAY frame (type=0x7) is used to initiate graceful shutdown of a connection by a server. Nginx sends GOAWAY frames when a keepalive connection needs to end (e.g. has hit `keepalive_timeout` or `keepalive_requests` limits). This happens regularly with default nginx http2 configs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.