cloudflare / cloudflare/cloudflared

🐛 QUIC-run tunnel hanging requests randomly

Open
#1,067 8 comments 0 reactions 0 assignees View on GitHub
Priority: Normal Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

Running cloudflared version 2023.8.2 (built 2023-08-31-1506 UTC) both client & server.

Tunnel ID: a9ee974a-f873-44c0-9b7a-2fb276b0d775
Tunnel using QUIC, no firewalls/limiters/DPIs. UDP buffer is extended w/ sysctl.

Cloudflared act as tcp proxy, requests would suddenly hang/timeout.

Do not know how to reproduce as it's happening randomly.

If I hit the tunnel directly (without cloudflared client), hanging response. Expected response is 200 OK blank response:
```
root@jkt02-ctn:~# curl https://-v
* Trying 2606:4700::6810:c723:443...
* TCP_NODELAY set
* Connected to (2606:4700::6810:c723) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* start date: Sep 8 08:32:36 2023 GMT
* expire date: Dec 7 08:32:35 2023 GMT
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x558354553300)
> GET / HTTP/2
> Host:
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!

```

No error from server side (current date 14th):
```
2023-09-11T23:24:42Z INF Unregistered tunnel connection connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-11T23:24:42Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-11T23:24:42Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-11T23:24:42Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-11T23:24:44Z WRN Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0
2023-09-11T23:25:00Z INF Registered tunnel connection connIndex=0 connection=68299612-c33d-48a8-83a4-6336b7291034 event=0 ip=2606:4700:a0::100 location=fra11 protocol=quic
2023-09-12T02:25:46Z INF Unregistered tunnel connection connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-12T02:25:46Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-12T02:25:46Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-12T02:25:46Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=2606:4700:a0::100
2023-09-12T02:25:48Z WRN Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0
2023-09-12T02:26:04Z INF Registered tunnel connection connIndex=0 connection=ae2d8e6f-8006-4f0f-bd44-d27dba090560 event=0 ip=2606:4700:a0::100 location=fra11 protocol=quic
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.