cloudflare / cloudflare/cloudflared

🐛 cloudflared grows without bounds when request rate exceeds upload bandwidth, destabilizing host

Open
#1,205 1 comment 1 reaction 0 assignees View on GitHub
Priority: Normal Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
When incoming http requests result in a requested data volume that exceeds the host's upload bandwidth, response bodies from the origin are buffered and queued to be copied out to the network. At the same time, new requests arrive, get proxied, the origin responds to them quickly, and their responses join the queue to be copied out. This continues without limit. This can be seen in the `cloudflared_tunnel_concurrent_requests_per_tunnel` metric, which goes from zero to over 10k.

Consequences:
- Cloudflared resident memory keeps growing. I've seen it go from several hundred MB to several gigabytes in minutes.
- It probably consumes an ever increasing number of goroutines and perhaps stacks, buffers, file descriptors.
- Eventually, clients that have been waiting for longer than 30 seconds start timing out and disconnecting, resulting in the agent logging a flood of `stream 3052609 canceled by remote with error code 0` errors.
- After a few minutes, the agent starts experiencing I/O errors on connections with the origin, e.g. `read tcp [::1]:34464->[::1]:443: read: connection reset by peer`. This is probably due to the memory and resource pressure on the host OS.
- In our particular case, cloudflared logs are shipped to the cloud (Datadog), and the high log volume (due to errors) exacerbates the upload bandwidth problem.
- The host becomes sluggish, and eventually unreponsive to remote management attempts

**To Reproduce**

This was experienced with Cloudflare Tunnel on multiple customer environments. We have a cloud-based process that requests a steady stream of data from customer's hosts. Customers sites that have ample bandwidth do fine. Sites that have constrained upload bandwidth experience this problem.

For the full details of several incidents, please refer to Cloudflare Support ticket 3133620.

**Expected behavior**

Cloudflared should provide an optional mechanism to protect itself (and the host) from this problem. One proposal is to have a new setting to limit the number of oustanding requests, which is already tracked with the `cloudflared_tunnel_concurrent_requests_per_tunnel` metric. When the limit is exceeded, the agent could respond to new requests with HTTP status 429 ("too many requests"). We implemented and successfully tested this in this PR in a forked repo: https://github.com/spotai/cloudflared/pull/1

**Environment and versions**
- OS: Linux / Ubuntu 22.04
- Architecture: amd64
- Version: 2024.2.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.