🐛Tunnel connection registered before remote config loaded

Recently, we switched from using local configuration files for cloudflared to managing configurations remotely via the Cloudflare Zero Trust dashboard.

After this change, we observed that upon startup, a cloudflared instance initially references the local config and begins establishing connections. A few seconds later, the remote configuration is loaded.

The issue is that as soon as the first connection is established, traffic is routed to cloudflared. Since the remote configuration is not yet loaded, there are no ingress rules, resulting in HTTP 503 errors for all incoming requests.

This flow looks evident in the logs:

> 2024-05-30T19:38:04Z INF Starting tunnel tunnelID=XXXXX
> 2024-05-30T19:38:04Z INF Version 2024.5.0
> 2024-05-30T19:38:04Z INF GOOS: linux, GOVersion: go1.22.2-devel-cf, GoArch: arm64
> 2024-05-30T19:38:04Z INF Settings: map[config:/etc/cloudflared/config/cloudflared-config.yaml cred-file:/etc/cloudflared/creds/credentials.json credentials-file:/etc/cloudflared/creds/credentials.json edge-ip-version:auto metrics:0.0.0.0:2000 no-autoupdate:true p:http2 protocol:http2]
> 2024-05-30T19:38:04Z INF Generated Connector ID: YYYYY
> 2024-05-30T19:38:04Z WRN No ingress rules were defined in provided config (if any) nor from the cli, cloudflared will return 503 for all incoming HTTP requests
> 2024-05-30T19:38:04Z INF Initial protocol http2
> 2024-05-30T19:38:04Z INF ICMP proxy will use 10.42.0.44 as source for IPv4
> 2024-05-30T19:38:04Z INF ICMP proxy will use fe30::e1f5:86ff:fe2d:45fb in zone eth0 as source for IPv6
> 2024-05-30T19:38:04Z INF Starting metrics server on [::]:2000/metrics
> 2024-05-30T19:38:04Z INF Registered tunnel connection connIndex=0 connection=4132692b-bd73-4aad-83d6-b029bd565440 event=0 ip=198.41.200.113 location=dme06 protocol=http2
> 2024-05-30T19:38:05Z INF Registered tunnel connection connIndex=1 connection=a32e9a6a-a6fd-43c5-947b-731d1e46df0f event=0 ip=198.41.192.167 location=dme05 protocol=http2
> 2024-05-30T19:38:06Z INF Registered tunnel connection connIndex=2 connection=2fb747ac-b9db-472c-9000-5b0aa025ecc event=0 ip=198.41.200.53 location=dme06 protocol=http2
> 2024-05-30T19:38:06Z INF Updated to new configuration config="{\"ingress\":[{\"service\":\"http://zzz-service:80\"}], \"warp-routing\":{\"enabled\":false}}" version=2
> 2024-05-30T19:38:07Z INF Registered tunnel connection connIndex=3 connection=dbc39486-4680-4a16-9631-331145988c3a event=0 ip=198.41.192.57 location=dme05 protocol=http2

We expect traffic to be routed to the cloudflared instance only after the remote configuration is fully loaded. Especially when no local ingress rules are defined!

I guess connections should be established or registered once the remote configuration has been applied to avoid concurrency issues.

This issue was encountered with the latest version of cloudflared: 2024.5.0.