cloudflare / cloudflare/cloudflared
Tunnel is in degraded state🐛
- Dominant language
- Go
- Stars
- 15.6k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
When I run cloudflared agent on windows or linux servers in Egypt or Lebanon, the status of theses tunnels are "Degraded". My tunnel successfuly connects to Marseille Edge but don't get connected to any other cloudflare edges (Zurich, Brussels, Paris ...)
**To Reproduce**
- Install cloudflared x64 on Windows using MSI
- Register cloudflared using the token
- Start cloudflared as a Windows service
If it's an issue with Cloudflare Tunnel:
4. Tunnel ID : 443133a6-ffb9-4e81-8d9f-1ce91ea8d224
5. cloudflared config: {\"noTLSVerify\":true},\"path\":\"apex\",\"service\":\"https://localhost:8443\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}} , tunnel is remotly managed.
**Expected behavior**
Tunnel to have a healthy status connected to two cloudflare datacenter edges.
**Environment and versions**
-cloudflared version 2023.6.1 (built 2023-06-20-1312 UTC), but is the same with older verision
-Windows Server 2019 , Build 1908, but it's the same with Linux boseS.
**Logs and errors**
`{"level":"info","tunnelID":"443133a6-ffb9-4e81-8d9f-1ce91ea8d224","time":"2023-07-06T12:27:59Z","message":"Starting tunnel"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"Version 2023.6.1"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"GOOS: windows, GOVersion: go1.19, GoArch: amd64"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"Settings: map[token:*****]"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"Environmental variables map[TUNNEL_LOGFILE:c:\\temp\\cloudflare.log TUNNEL_LOGLEVEL:debug TUNNEL_PROTO_LOGLEVEL:debug TUNNEL_TRANSPORT_LOGLEVEL:debug TUNNEL_TRANSPORT_PROTOCOL:http2]"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"cloudflared will not automatically update on Windows systems."}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"Generated Connector ID: 7cac4633-b07e-4ff8-829c-473924a86c83"}
{"level":"debug","time":"2023-07-06T12:27:59Z","message":"Fetched protocol: quic"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"Initial protocol http2"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"ICMP proxy will use 10.224.128.26 as source for IPv4"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"ICMP proxy will use ::1 in zone Loopback Pseudo-Interface 1 as source for IPv6"}
{"level":"debug","event":0,"event":0,"domain":"_v2-origintunneld._tcp.argotunnel.com","time":"2023-07-06T12:27:59Z","message":"edge discovery: looking up edge SRV record"}
{"level":"debug","event":0,"addresses":["198.41.192.67","198.41.192.37","198.41.192.227","198.41.192.47","198.41.192.77","198.41.192.167","198.41.192.27","198.41.192.107","198.41.192.57","198.41.192.7"],"time":"2023-07-06T12:27:59Z","message":"edge discovery: resolved edge addresses"}
{"level":"debug","event":0,"addresses":["198.41.200.43","198.41.200.33","198.41.200.233","198.41.200.13","198.41.200.113","198.41.200.73","198.41.200.23","198.41.200.53","198.41.200.63","198.41.200.193"],"time":"2023-07-06T12:27:59Z","message":"edge discovery: resolved edge addresses"}
{"level":"debug","event":0,"event":0,"domain":"_v2-origintunneld._tcp.argotunnel.com","time":"2023-07-06T12:27:59Z","message":"edge discovery: looking up edge SRV record"}
{"level":"info","time":"2023-07-06T12:27:59Z","message":"Starting metrics server on 127.0.0.1:54518/metrics"}
{"level":"debug","event":0,"addresses":["198.41.192.67","198.41.192.37","198.41.192.227","198.41.192.47","198.41.192.77","198.41.192.167","198.41.192.27","198.41.192.107","198.41.192.57","198.41.192.7"],"time":"2023-07-06T12:27:59Z","message":"edge discovery: resolved edge addresses"}
{"level":"debug","event":0,"addresses":["198.41.200.43","198.41.200.33","198.41.200.233","198.41.200.13","198.41.200.113","198.41.200.73","198.41.200.23","198.41.200.53","198.41.200.63","198.41.200.193"],"time":"2023-07-06T12:27:59Z","message":"edge discovery: resolved edge addresses"}
{"level":"debug","connIndex":0,"event":0,"ip":"198.41.192.107","time":"2023-07-06T12:27:59Z","message":"edge discovery: giving new address to connection"}
{"level":"debug","event":0,"ip":"198.41.192.107","connIndex":0,"time":"2023-07-06T12:27:59Z","message":"Connecting via http2"}
{"level":"info","event":0,"connection":"2373ca5c-7841-45f9-91ee-1395be8460c1","connIndex":0,"location":"MRS","ip":"198.41.192.107","protocol":"http2","time":"2023-07-06T12:27:59Z","message":"Registered tunnel connection"}
{"level":"debug","connIndex":1,"event":0,"ip":"198.41.200.43","time":"2023-07-06T12:27:59Z","message":"edge discovery: giving new address to connection"}
{"level":"debug","event":0,"ip":"198.41.200.43","connIndex":1,"time":"2023-07-06T12:27:59Z","message":"Connecting via http2"}
{"level":"info","event":0,"connection":"a51a7ab8-1fe2-4d87-aeee-8a6fd590c178","connIndex":1,"location":"MRS","ip":"198.41.200.43","protocol":"http2","time":"2023-07-06T12:28:00Z","message":"Registered tunnel connection"}
{"level":"info","time":"2023-07-06T12:28:00Z","message":"cloudflared does not support loading the system root certificate pool on Windows. Please use --origin-ca-pool to specify the path to the certificate pool"}
{"level":"info","version":2,"config":"{\"ingress\":[{\"hostname\":\"removed.domain.tld\",\"originRequest\":{\"noTLSVerify\":true},\"path\":\"apex\",\"service\":\"https://localhost:8443\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}","time":"2023-07-06T12:28:00Z","message":"Updated to new configuration"}
{"level":"debug","connIndex":2,"event":0,"ip":"198.41.200.53","time":"2023-07-06T12:28:00Z","message":"edge discovery: giving new address to connection"}
{"level":"debug","event":0,"ip":"198.41.200.53","connIndex":2,"time":"2023-07-06T12:28:00Z","message":"Connecting via http2"}
`
**Additional context**
When I force region to be "US" using --region US I am connected correctly to two cloudflare edge location (ORD & IAD). However it seems not supported to force to use any other regions than US (Asia, Europe , Whatever ....)
`2023-07-06T12:38:05Z INF Starting tunnel tunnelID=443133a6-ffb9-4e81-8d9f-1ce91ea8d224
2023-07-06T12:38:05Z INF Version 2023.6.1
2023-07-06T12:38:05Z INF GOOS: windows, GOVersion: go1.19, GoArch: amd64
2023-07-06T12:38:05Z INF Settings: map[region:US token:*****]
2023-07-06T12:38:05Z INF Environmental variables map[TUNNEL_LOGFILE:c:\temp\cloudflare.log TUNNEL_LOGLEVEL:debug TUNNEL_PROTO_LOGLEVEL:debug TUNNEL_TRANSPORT_LOGLEVEL:debug TUNNEL_TRANSPORT_PROTOCOL:http2]
2023-07-06T12:38:05Z INF cloudflared will not automatically update on Windows systems.
2023-07-06T12:38:05Z INF Generated Connector ID: decd56cc-22e4-4538-bf3f-15d9c5e8b26f
2023-07-06T12:38:05Z DBG Fetched protocol: quic
2023-07-06T12:38:05Z INF Initial protocol http2
2023-07-06T12:38:05Z INF ICMP proxy will use 10.224.128.26 as source for IPv4
2023-07-06T12:38:05Z INF ICMP proxy will use ::1 in zone Loopback Pseudo-Interface 1 as source for IPv6
2023-07-06T12:38:05Z DBG edge discovery: looking up edge SRV record domain=_US-v2-origintunneld._tcp.argotunnel.com event=0
2023-07-06T12:38:05Z DBG edge discovery: resolved edge addresses addresses=["198.41.218.3","198.41.218.8","198.41.218.10","198.41.218.2","198.41.218.7","198.41.218.9","198.41.218.5","198.41.218.6","198.41.218.1","198.41.218.4"] event=0
2023-07-06T12:38:05Z DBG edge discovery: resolved edge addresses addresses=["198.41.219.10","198.41.219.9","198.41.219.2","198.41.219.8","198.41.219.1","198.41.219.7","198.41.219.5","198.41.219.4","198.41.219.6","198.41.219.3"] event=0
2023-07-06T12:38:05Z INF Starting metrics server on 127.0.0.1:54525/metrics
2023-07-06T12:38:05Z DBG edge discovery: looking up edge SRV record domain=_US-v2-origintunneld._tcp.argotunnel.com event=0
2023-07-06T12:38:05Z DBG edge discovery: resolved edge addresses addresses=["198.41.218.3","198.41.218.8","198.41.218.10","198.41.218.2","198.41.218.7","198.41.218.9","198.41.218.5","198.41.218.6","198.41.218.1","198.41.218.4"] event=0
2023-07-06T12:38:05Z DBG edge discovery: resolved edge addresses addresses=["198.41.219.10","198.41.219.9","198.41.219.2","198.41.219.8","198.41.219.1","198.41.219.7","198.41.219.5","198.41.219.4","198.41.219.6","198.41.219.3"] event=0
2023-07-06T12:38:05Z DBG edge discovery: giving new address to connection connIndex=0 event=0 ip=198.41.218.8
2023-07-06T12:38:06Z DBG Connecting via http2 connIndex=0 event=0 ip=198.41.218.8
2023-07-06T12:38:06Z INF Registered tunnel connection connIndex=0 connection=fdbffc03-4d9b-4a21-a386-1e9deb5bf605 event=0 ip=198.41.218.8 location=ORD protocol=http2
2023-07-06T12:38:06Z DBG edge discovery: giving new address to connection connIndex=1 event=0 ip=198.41.219.9
2023-07-06T12:38:07Z DBG Connecting via http2 connIndex=1 event=0 ip=198.41.219.9
2023-07-06T12:38:07Z INF cloudflared does not support loading the system root certificate pool on Windows. Please use --origin-ca-pool to specify the path to the certificate pool
2023-07-06T12:38:07Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"removed.domain.tld\",\"originRequest\":{\"noTLSVerify\":true},\"path\":\"apex\",\"service\":\"https://localhost:8443\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=2
2023-07-06T12:38:07Z DBG edge discovery: giving new address to connection connIndex=2 event=0 ip=198.41.219.2
2023-07-06T12:38:07Z INF Registered tunnel connection connIndex=1 connection=546dde2c-a76d-4c4b-ac07-ecda9c4237b6 event=0 ip=198.41.219.9 location=IAD protocol=http2
2023-07-06T12:38:08Z DBG Connecting via http2 connIndex=2 event=0 ip=198.41.219.2
2023-07-06T12:38:08Z INF Registered tunnel connection connIndex=2 connection=a4828a5a-2983-4f79-ab32-56104e2ea6d8 event=0 ip=198.41.219.2 location=IAD protocol=http2
2023-07-06T12:38:08Z DBG edge discovery: giving new address to connection connIndex=3 event=0 ip=198.41.218.1
2023-07-06T12:38:09Z DBG Connecting via http2 connIndex=3 event=0 ip=198.41.218.1
2023-07-06T12:38:09Z INF Registered tunnel connection connIndex=3 connection=cf58b4c6-3332-4d7c-9187-23d909277a6e event=0 ip=198.41.218.1 location=ORD protocol=http2`
One interesting information to share is the DNS resolution from my server :
```
PS C:\> nslookup -q=srv _v2-origintunneld._tcp.argotunnel.com
Server: UnKnown
Address: 10.224.129.2
Non-authoritative answer:
_v2-origintunneld._tcp.argotunnel.com SRV service location:
priority = 2
weight = 1
port = 7844
svr hostname = region2.v2.argotunnel.com
_v2-origintunneld._tcp.argotunnel.com SRV service location:
priority = 1
weight = 1
port = 7844
svr hostname = region1.v2.argotunnel.com
region2.v2.argotunnel.com internet address = 198.41.200.193
region2.v2.argotunnel.com internet address = 198.41.200.43
region2.v2.argotunnel.com internet address = 198.41.200.33
region2.v2.argotunnel.com internet address = 198.41.200.233
region2.v2.argotunnel.com internet address = 198.41.200.13
region2.v2.argotunnel.com internet address = 198.41.200.113
region2.v2.argotunnel.com internet address = 198.41.200.73
region2.v2.argotunnel.com internet address = 198.41.200.23
region2.v2.argotunnel.com internet address = 198.41.200.53
region2.v2.argotunnel.com internet address = 198.41.200.63
region2.v2.argotunnel.com AAAA IPv6 address = 2606:4700:a8::10
region2.v2.argotunnel.com AAAA IPv6 address = 2606:4700:a8::2
region2.v2.argotunnel.com AAAA IPv6 address = 2606:4700:a8::8
region2.v2.argotunnel.com AAAA IPv6 address = 2606:4700:a8::7
region2.v2.argotunnel.com AAAA IPv6 address = 2606:4700:a8::5
region2.v2.argotunnel.com AAAA IPv6 address = 2606:4700:a8::1
region2.v2.argotunnel.com AAAA IPv6 address = 2606:4700:a8::6
```
Some ports are open to region1 and region2
```
TCP 10.224.128.26:54534 198.41.192.167:7844 ESTABLISHED
TCP 10.224.128.26:54535 198.41.200.233:7844 ESTABLISHED
TCP 10.224.128.26:54536 198.41.200.33:7844 ESTABLISHED
TCP 10.224.128.26:54537 198.41.192.77:7844 ESTABLISHED
```
Contributor guide
Assessment
This issue has not been assessed yet.