users reporting tunnel failures

Users seeing the "cloudflare gateway error" page, intermittently today, when using a tunnel to a web site on AWS.

I then allowed UDP outbound from the ec2 instance, but still seeing a lot of errors in the logs:

Apr 26 11:06:50 jenkins-sys.myco.com cloudflared[978]: If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=1
Apr 26 11:06:50 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T11:06:50Z INF Switching to fallback protocol http2 connIndex=1
Apr 26 11:06:51 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T11:06:51Z INF Connection d2838bf9-b03d-4fcd-af18-1b62d2372abe registered connIndex=1 location=LAX

(restarting because I allowed outbound UDP)

Apr 26 13:19:06 jenkins-sys.myco.com systemd[1]: Stopping cloudflared...
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T13:19:06Z INF Initiating graceful shutdown due to signal terminated ...
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T13:19:06Z INF Unregistered tunnel connection connIndex=0
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T13:19:06Z INF Unregistered tunnel connection connIndex=2
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T13:19:06Z INF Unregistered tunnel connection connIndex=1
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T13:19:06Z INF Unregistered tunnel connection connIndex=3
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T13:19:06Z INF Tunnel server stopped
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[978]: 2022-04-26T13:19:06Z INF Metrics server stopped
Apr 26 13:19:06 jenkins-sys.myco.com systemd[1]: Stopped cloudflared.
Apr 26 13:19:06 jenkins-sys.myco.com systemd[1]: Starting cloudflared...
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Starting tunnel tunnelID=dd3d43a3-2923-40b9-ab88-9d538025aad9
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Version 2022.4.1
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF GOOS: linux, GOVersion: go1.17.5, GoArch: amd64
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Settings: map[no-autoupdate:true token:*****]
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Generated Connector ID: 04a10e73-ad90-498e-8d97-dfc9ea609874
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF cloudflared will not automatically update if installed by a package manager.
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Initial protocol quic
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:06Z INF Starting metrics server on 127.0.0.1:33629/metrics
Apr 26 13:19:06 jenkins-sys.myco.com cloudflared[29131]: 2022/04/26 13:19:06 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Apr 26 13:19:07 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:07Z INF Connection 9726d9a3-17ee-422e-b583-fd0464996e80 registered connIndex=0 location=PDX
Apr 26 13:19:07 jenkins-sys.myco.com systemd[1]: Started cloudflared.
Apr 26 13:19:07 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:07Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"jenkins-sys.xxx.com\", \"originRequest\":{\"httpHostHeader\":\"\", \"noTLSVerify\":true}, \"service\":\"http://localhost:8080\"}, {\"service\":\"http_status:404\"}], \"warp-routing\":{\"enabled\":false}}" version=2
Apr 26 13:19:07 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:07Z INF Connection 53ee58f4-6af9-4960-925d-b6524d37c5af registered connIndex=1 location=DEN
Apr 26 13:19:08 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:08Z WRN Failed to serve quic connection error="already connected to this server, trying another address" connIndex=2
Apr 26 13:19:08 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:08Z WRN Unable to establish connection. error="already connected to this server, trying another address" connIndex=2
Apr 26 13:19:09 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:09Z INF Connection 2a27ea7d-dbb8-4a06-855f-fda17fbf1520 registered connIndex=3 location=LAX
Apr 26 13:19:10 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:10Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=2
Apr 26 13:19:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:19:19Z INF Connection 4a5c2467-d904-48f5-95cf-6cb7132b6191 registered connIndex=2 location=PDX
Apr 26 13:36:43 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:36:43Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" cfRay=701fb5816b27a3b5-MRS ingressRule=0 originService=http://localhost:8080
Apr 26 13:36:43 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:36:43Z ERR Failed to handle QUIC stream error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" connIndex=2
Apr 26 13:57:21 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:57:21Z INF Unregistered tunnel connection connIndex=1
Apr 26 13:57:21 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:57:21Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1
Apr 26 13:57:21 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:57:21Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1
Apr 26 13:57:21 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:57:21Z INF Retrying connection in up to 1s seconds connIndex=1
Apr 26 13:57:23 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:57:23Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
Apr 26 13:57:23 jenkins-sys.myco.com cloudflared[29131]: If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=1
Apr 26 13:57:23 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:57:23Z INF Switching to fallback protocol http2 connIndex=1
Apr 26 13:57:23 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T13:57:23Z INF Connection 5c313303-b31f-4cbe-96d7-3c111c8ac630 registered connIndex=1 location=DEN
Apr 26 14:21:59 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T14:21:59Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" cfRay=701ff7ce7840a3af-MRS ingressRule=0 originService=http://localhost:8080
Apr 26 14:21:59 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T14:21:59Z ERR Failed to handle QUIC stream error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" connIndex=2
Apr 26 15:11:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:11:17Z INF Lost connection with the edge connIndex=1
Apr 26 15:11:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:11:17Z WRN Serve tunnel error error="connection with edge closed" connIndex=1
Apr 26 15:11:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:11:17Z INF Retrying connection in up to 1s seconds connIndex=1
Apr 26 15:11:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:11:17Z INF Unregistered tunnel connection connIndex=1
Apr 26 15:11:18 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:11:18Z INF Changing protocol to quic connIndex=1
Apr 26 15:11:18 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:11:18Z INF Connection 56c8588c-f500-488c-a68a-7963d0afcb64 registered connIndex=1 location=DEN
Apr 26 15:43:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:43:19Z INF Unregistered tunnel connection connIndex=1
Apr 26 15:43:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:43:19Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=1
Apr 26 15:43:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:43:19Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=1
Apr 26 15:43:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:43:19Z INF Retrying connection in up to 1s seconds connIndex=1
Apr 26 15:43:20 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:43:20Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
Apr 26 15:43:20 jenkins-sys.myco.com cloudflared[29131]: If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=1
Apr 26 15:43:20 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:43:20Z INF Switching to fallback protocol http2 connIndex=1
Apr 26 15:43:20 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T15:43:20Z INF Connection ddfe37d1-bc11-4ccb-aef3-8e9444d3e010 registered connIndex=1 location=LAX
Apr 26 16:26:18 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:26:18Z INF Lost connection with the edge connIndex=1
Apr 26 16:26:18 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:26:18Z WRN Serve tunnel error error="connection with edge closed" connIndex=1
Apr 26 16:26:18 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:26:18Z INF Retrying connection in up to 1s seconds connIndex=1
Apr 26 16:26:18 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:26:18Z INF Unregistered tunnel connection connIndex=1
Apr 26 16:26:20 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:26:20Z INF Changing protocol to quic connIndex=1
Apr 26 16:26:21 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:26:21Z INF Connection 9c341b63-0be9-472b-8391-fa2f0db0b457 registered connIndex=1 location=DEN
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:27:51Z INF Unregistered tunnel connection connIndex=1
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:27:51Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:27:51Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:27:51Z INF Retrying connection in up to 1s seconds connIndex=1
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:27:51Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=1
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:27:51Z INF Switching to fallback protocol http2 connIndex=1
Apr 26 16:27:51 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:27:51Z INF Connection 2f9a037f-a5d7-4894-9eb5-3a3684c7531a registered connIndex=1 location=DEN
Apr 26 16:28:31 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:28:31Z INF Lost connection with the edge connIndex=1
Apr 26 16:28:31 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:28:31Z WRN Serve tunnel error error="connection with edge closed" connIndex=1
Apr 26 16:28:31 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:28:31Z INF Retrying connection in up to 1s seconds connIndex=1
Apr 26 16:28:31 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:28:31Z INF Unregistered tunnel connection connIndex=1
Apr 26 16:28:33 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:28:33Z INF Changing protocol to quic connIndex=1
Apr 26 16:28:33 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:28:33Z INF Connection 429c997a-e7b9-4959-8369-d08abc8e790d registered connIndex=1 location=LAX
Apr 26 16:31:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:31:17Z INF Unregistered tunnel connection connIndex=3
Apr 26 16:31:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:31:17Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3
Apr 26 16:31:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:31:17Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3
Apr 26 16:31:17 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:31:17Z INF Retrying connection in up to 1s seconds connIndex=3
Apr 26 16:31:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:31:19Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
Apr 26 16:31:19 jenkins-sys.myco.com cloudflared[29131]: If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=3
Apr 26 16:31:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:31:19Z INF Switching to fallback protocol http2 connIndex=3
Apr 26 16:31:19 jenkins-sys.myco.com cloudflared[29131]: 2022-04-26T16:31:19Z INF Connection 2853a71e-8367-4a55-b679-5196e412d4b9 registered connIndex=3 location=DEN