cloudflare / cloudflare/cloudflared
🐛Origin dial fails instantly with i/o timeout when originRequest.connectTimeout is set via remote-managed tunnel config
- Dominant language
- Go
- Stars
- 15.6k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
When a tunnel's ingress config is managed remotely (`config_src: cloudflare`, i.e. pushed via the API / Terraform's `cloudflare_zero_trust_tunnel_cloudflared_config` / the dashboard) and an ingress rule's `originRequest.connectTi
meout` is set to any explicit value, every dial to that origin fails instantly with `dial tcp: lookup : i/o timeout` — even though the origin is up and reachable the whole time.
The failure happens in well under 200ms, nowhere near any configured timeout, and `strace -f -e trace=network` on the `cloudflared` process shows **zero `socket()`/`connect()` syscalls** for the origin during the failed request —
the dial never reaches the OS network stack. This points to the dial's `context.Context` already being expired/cancelled before `net.Dialer` is invoked, consistent with a unit or sign error converting `connectTimeout` (documente
d as nanoseconds) into a deadline.
Removing `connectTimeout` from `originRequest` entirely (so cloudflared falls back to its own default) fixes it immediately, with no other change.
**To Reproduce**
Steps to reproduce the behavior:
1. Configure a named tunnel with `config_src: cloudflare` and an ingress rule with an explicit `originRequest.connectTimeout`, e.g.:
```json
{
"hostname": "app.example.com",
"service": "http://origin-host:8080",
"originRequest": {
"connectTimeout": 10000000000,
"httpHostHeader": "app.example.com"
}
}
```
(`10000000000` = 10s expressed in nanoseconds, per the documented unit for this field.)
2. Run `cloudflared tunnel run` with the origin actually up and reachable.
3. Request the hostname through the tunnel — see `dial tcp: lookup : i/o timeout` instantly, every time.
4. Tunnel ID: `fbc8884a-aa1e-468a-b3bd-c93782598912`
5. cloudflared config (as pushed via the API, redirect/hostname redacted for this writeup but reproducible with any hostname/origin pair):
```json
{
"ingress": [
{
"hostname": "echo-basic.zt.example.com",
"originRequest": {
"connectTimeout": 10000000000,
"httpHostHeader": "echo-basic.zt.example.com"
},
"service": "http://echo-basic:8080"
},
{ "service": "http_status:404" }
],
"warp-routing": { "enabled": false }
}
```
**Expected behavior**
The origin is dialed with (approximately) a 10-second connect timeout, and the request succeeds since the origin responds immediately.
**Environment and versions**
- OS: Linux (WSL2, Ubuntu), also reproduced with `cloudflared` running as a bare host process outside any container
- Architecture: AMD64
- Version: reproduced on both **2026.7.3** (go1.26.4) and **2026.2.0** (go1.24.13) — not a recent regression
**Logs and errors**
```
INF Registered tunnel connection connIndex=0 connection=... protocol=quic
INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"echo-basic.zt.example.com\",\"originRequest\":{\"connectTimeout\":10000000000,\"httpHostHeader\":\"echo-basic.zt.example.com\"},\"service\":\"http://echo-basi
c:8080\"},...],\"warp-routing\":{\"enabled\":false}}" version=5
ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup echo-basic: i/o timeout" connIndex=0 event=1 ingressRule=0 originService=http://ech
o-basic:8080
ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup echo-basic: i/o timeout" connIndex=0 dest=https://echo-basic.zt.examp
le.com/health event=0 ip=198.41.192.167 type=http
```
`strace -f -tt -e trace=network` over the same window captures zero `socket()`/`connect()` calls referencing the origin hostname/IP/port at all — the process-level rejection happens before any syscall.
**Additional context**
- Reproduced in three different contexts, ruling out Docker/network-driver causes:
1. `cloudflared` inside a `docker compose` service, origin containers on the same user-defined bridge network, addressed by container hostname.
2. Same setup, origin addressed by raw container IP instead of hostname (same failure, so not DNS-specific).
3. `cloudflared` run as a bare process directly on the host (no Docker at all), origin addressed via `localhost:` — same instant failure, including for `localhost` itself.
- A separate `nicolaka/netshoot` container/process on the identical network resolves and connects to the same origin (`ping`, `nslookup`, `nc -vz`, `curl`) instantly and reliably throughout every failed `cloudflared` attempt, rul
ing out the origin or network path.
- The same `cloudflared` binary, copied into that working container and run standalone against the same tunnel/config, fails identically — ruling out the container image/base OS as a factor.
- `--max-active-flows ` (appears to gate WARP private-network routing flows, not ordinary ingress) does **not** affect this — tested explicitly, no effect.
- Origins specified locally rather than through remote config (`cloudflared tunnel --url http://origin:port`, i.e. quick tunnels / local config, which never set `connectTimeout` this way) were not observed to have this problem —
the bug appears specific to how `connectTimeout` is parsed/applied when it arrives via the remotely-fetched ingress config.
- Workaround: omit `originRequest.connectTimeout` from the ingress rule entirely.
Contributor guide
Research direction
Start by reproducing the remote-managed ingress case with an explicit connectTimeout, then trace how that value is parsed and passed to net.Dialer. Compare it with a configuration that omits connectTimeout and with locally specified origins. Done means the origin receives a real dial attempt and an explicit 10-second timeout no longer causes an immediate failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100