cloudflare / cloudflare/cloudflared

🐛 Quic is has slower response times than http2 protocol (confirmed issue with cloudflare support)

Open
#895 9 comments 19 reactions 0 assignees View on GitHub
Priority: Normal Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

quic is **default** for cloudflared but it has performance issues as observed in our perf tests and also confirmed by Cloudflare support. The purpose of this ticket is raise this issue in the open and allow a way for the community to keep track of this. It cost us a week of communications with support before it was found that it was a quic vs http2 issue. We had to switch to using http2.

For anyone internal triaging this, the ticket is at https://support.cloudflare.com/hc/en-us/requests/2690173

**To Reproduce**
Steps to reproduce the behavior:

1. We are using cloudflare/cloudflared:2023.2.1-arm64
2. Create a k8s cluster -- we are testing on GKE.
3. Follow the steps at https://developers.cloudflare.com/cloudflare-one/tutorials/many-cfd-one-tunnel/ except that we are using the dashboard to create the mapping. Therefore, we use a TUNNEL_TOKEN.
4. For our perf, we set up a deployment using the httpbin docker image.
5. We ran a bunch of tests against the httpbin deployment against the `stream/100` endpoint using k6.io using both http2 and quic.
6. We noticed that overall http2 was much faster in terms of p95 response times. I've attached the graphs

Screenshot 2023-01-30 at 9 44 46 AM

Screenshot 2023-01-30 at 9 44 50 AM

**Additional context**
Add any other context about the problem here.

```
Our Tunnels engineering manager looked at the escalation ticket and he confirmed that using QUIC will be lower than using http/2 as you also noted.

In the case of QUIC vs HTTP2, it is important to note that QUIC transport operates at user space, so it is normal to be more demanding in terms of CPU.

So, if you compare both with the same restrictions, then it is expected for performance to be slower.

Our EM is going to do some benchmark testing next quarter with an eye towards seeing if there are QUIC improvements that can be made.
```

For now, we are going to set TUNNEL_TRANSPORT_PROTOCOL to http2.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.