cloudflare / cloudflare/cloudflared
🐛 GOAWAY frames not handled correctly with http2
- Dominant language
- Go
- Stars
- 15.6k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
When connecting to nginx with https using http2 the following error appears in the logs intermittently and results in client facing 502 errors;
`Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error`
**To Reproduce**
Steps to reproduce the behavior:
1. Install nginx container (such as nginx:alpine) and cloudflared:latest
2. Use an nginx config file with http2 on (Using a short timeout or request limit triggers GOAWAY frames faster);
```
server {
listen 443 ssl default_server;
http2 on;
keepalive_timeout 1s;
keepalive_requests 1;
}
```
3. Use a cloudflared config file with http2 on;
```
ingress:
- service: https://nginx:443
originRequest:
http2Origin: true
```
4. Start sending requests through cloudflared and view logs. You will see the error;
```
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
```
5. Setting `http2Origin: false` resolves the issue.
**Expected behavior**
Cloudflared should see the GOAWAY frame and retry the request.
**Environment and versions**
- Cloudflared Version: 2024.6.0
**Additional context**
Appears to be an issue with `http.Request` missing a no-op `GetBody()` method.
The GOAWAY frame (type=0x7) is used to initiate graceful shutdown of a connection by a server. Nginx sends GOAWAY frames when a keepalive connection needs to end (e.g. has hit `keepalive_timeout` or `keepalive_requests` limits). This happens regularly with default nginx http2 configs.
Contributor guide
Assessment
This issue has not been assessed yet.