cloudflare / cloudflare/cloudflared

🐛 502 HTTP errors on large number of JS files

Open
#774 0 comments 0 reactions 0 assignees View on GitHub
Priority: Normal Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
We are using cloudflared tunnels to reach some internal services secured using Zero trust and Access applications.
One of them is Kibana running with ECK.
Randomly we fail to load kibana UI which is secured using a kuberntes deployed cloudflare tunnel on the same cluster then Kibana.
Some JS files are impossible to load leading to 502.

If I use a port-forward eveyrthing is loading fine !
on the other side if I use a cloudflare tunnel directly deployed on EC2 everything is fine. So my theory is that a cloudflared deployed on Kubernetes can have some internal issues. But even using the tracing loglevel impossible to have a clear reason :
```
2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Request content length 0
2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Status: 200 OK served by ingress 8
2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Response Headers map[Cache-Control:[max-age=31536000] Connection:[keep-alive] Content-Encoding:[gzip] Content-Type:[application/javascript; charset=utf-8] Date:[Fri, 07 Oct 2022 12:05:06 GMT] Kbn-License-Sig:[0bbc5af556468fe9fe09e1957c8f92785ed89b0239b47bc58124941e84cf5a83] Kbn-Name:[kube-logging] Keep-Alive:[timeout=120] Referrer-Policy:[no-referrer-when-downgrade] Vary:[accept-encoding] X-Content-Type-Options:[nosniff]]
2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Response content length unknown
```

But we got some 502 on web browser side.

**To Reproduce**
Steps to reproduce the behavior:
1. Configure a ECK stack and try to reach the kibana UI
2. Deploy a cloudflared tunnel to on Kubernetes configured to access Kibana UI

If it's an issue with Cloudflare Tunnel:
4. Tunnel ID : `9e216a1f-1828-4e91-8bdd-0c3a50ddb772`
5. cloudflared config:
```
- hostname: logs.internal.mydomain.co
service: https://kube-logging-kb-http.kube-logging:5601
originRequest:
noTLSVerify: true
```

**Expected behavior**
We should be able to load the Kibana UI without any issue

**Environment and versions**
- OS: Debian GNU/Linux 11
- Architecture: AMD
- Version: 2022.10.0

**Logs and errors**
502 errors when using cloudflared kubernetes pod tunnel.

**Additional context**
`CF-RAY: 756680c7b851d2f5-CDG`

I've tried to set those setting on the clouflared kubernetes pod but nothing change.
```
sysctl -w net.core.rmem_max=4000000
sysctl -w net.core.somaxconn=4096
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.