tailscale / tailscale/tailscale

logtail, bootstrapDNS too aggressive when Internet's known down

Open
#3,474 1 comment 0 reactions 1 assignee Claimed by @bradfitz View on GitHub
needs-fix
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
1d 23h
Merged PRs (30d)
132

Description

Log snippet from #3473, it looks like we're trying bootstrapDNS DNS lookups when the internet is already known down.

Or maybe the bootstrap lookup started before we found the internet to be down and we just didn't abort the in-progress DNS lookup.

```
Dec 01 21:50:25 eru tailscaled[30516]: LinkChange: all links down; pausing: interfaces.State{defaultRoute= ifs={tailscale0:[100.123.106.4/32 fd7a:115c:a1e0:ab12:4843:cd96:627b:6a04/128]} v4=false v6=false}
Dec 01 21:50:25 eru tailscaled[30516]: magicsock: SetNetworkUp(false)
Dec 01 21:50:25 eru tailscaled[30516]: magicsock: closing connection to derp-10 (network-down), age 3m0s
Dec 01 21:50:25 eru tailscaled[30516]: magicsock: 0 active derp conns
Dec 01 21:50:25 eru tailscaled[30516]: dns: Set: {DefaultResolvers:[] Routes:{liftoff.io.:[172.20.0.2:53]} SearchDomains:[liftoff.io.] Hosts:3}
Dec 01 21:50:25 eru tailscaled[30516]: dns: Resolvercfg: {Routes:{} Hosts:3 LocalDomains:[]}
Dec 01 21:50:25 eru tailscaled[30516]: control: mapRoutine: paused
Dec 01 21:50:25 eru tailscaled[30516]: control: mapRoutine: awaiting unpause
Dec 01 21:50:25 eru tailscaled[30516]: dns: OScfg: {Nameservers:[172.20.0.2] SearchDomains:[liftoff.io.] MatchDomains:[liftoff.io.]}
Dec 01 21:50:25 eru tailscaled[30516]: health("overall"): error: network down
Dec 01 21:50:25 eru tailscaled[30516]: wgengine: set DNS config again after major link change
Dec 01 21:50:25 eru tailscaled[30516]: magicsock: endpoints changed: 127.0.0.1:41641 (local), [::1]:41641 (local)
Dec 01 21:50:25 eru tailscaled[30516]: control: client.newEndpoints(0, [127.0.0.1:41641 [::1]:41641])
Dec 01 21:50:25 eru tailscaled[30516]: LinkChange: all links down; pausing: interfaces.State{defaultRoute= ifs={tailscale0:[100.123.106.4/32 fd7a:115c:a1e0:ab12:4843:cd96:627b:6a04/128]} v4=false v6=false}
Dec 01 21:50:29 eru tailscaled[30516]: LinkChange: all links down; pausing: interfaces.State{defaultRoute= ifs={tailscale0:[100.123.106.4/32 fd7a:115c:a1e0:ab12:4843:cd96:627b:6a04/128]} v4=false v6=false}
Dec 01 21:50:29 eru tailscaled[30516]: LinkChange: all links down; pausing: interfaces.State{defaultRoute= ifs={tailscale0:[100.123.106.4/32 fd7a:115c:a1e0:ab12:4843:cd96:627b:6a04/128]} v4=false v6=false}
Dec 01 21:50:30 eru tailscaled[30516]: logtail: dial "log.tailscale.io:443" failed: dial tcp: lookup log.tailscale.io on 127.0.0.53:53: server misbehaving (in 5.005s), trying bootstrap...
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp9c.tailscale.com", "155.138.243.219") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp9c.tailscale.com", "155.138.243.219") for "log.tailscale.io" error: Get "https://derp9c.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp 155.138.243.219:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp8d.tailscale.com", "2a03:b0c0:1:d0::e08:e001") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp8d.tailscale.com", "2a03:b0c0:1:d0::e08:e001") for "log.tailscale.io" error: Get "https://derp8d.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp [2a03:b0c0:1:d0::e08:e001]:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp9.tailscale.com", "207.148.3.137") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp9.tailscale.com", "207.148.3.137") for "log.tailscale.io" error: Get "https://derp9.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp 207.148.3.137:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.io" error: Get "https://derp12b.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp [2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f]:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp3.tailscale.com", "68.183.179.66") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp3.tailscale.com", "68.183.179.66") for "log.tailscale.io" error: Get "https://derp3.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp 68.183.179.66:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.io" error: Get "https://derp12.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp [2001:19f0:5c01:289:5400:3ff:fe8d:cb5e]:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp10.tailscale.com", "137.220.36.168") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp10.tailscale.com", "137.220.36.168") for "log.tailscale.io" error: Get "https://derp10.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp 137.220.36.168:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp2.tailscale.com", "2604:a880:2:d1::c5:7001") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp2.tailscale.com", "2604:a880:2:d1::c5:7001") for "log.tailscale.io" error: Get "https://derp2.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp [2604:a880:2:d1::c5:7001]:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.io" error: Get "https://derp7.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp 167.179.89.145:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.io" error: Get "https://derp12b.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp [2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f]:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp4d.tailscale.com", "134.122.94.167") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp4d.tailscale.com", "134.122.94.167") for "log.tailscale.io" error: Get "https://derp4d.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp 134.122.94.167:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: trying bootstrapDNS("derp2b.tailscale.com", "2604:a880:4:1d0::29:9000") for "log.tailscale.io" ...
Dec 01 21:50:30 eru tailscaled[30516]: bootstrapDNS("derp2b.tailscale.com", "2604:a880:4:1d0::29:9000") for "log.tailscale.io" error: Get "https://derp2b.tailscale.com/bootstrap-dns?q=log.tailscale.io": dial tcp [2604:a880:4:1d0::29:9000]:443: connect: network is unreachable
Dec 01 21:50:30 eru tailscaled[30516]: logtail: internet down; waiting
Dec 01 21:50:30 eru tailscaled[30516]: LinkChange: all links down; pausing: interfaces.State{defaultRoute= ifs={tailscale0:[100.123.106.4/32 fd7a:115c:a1e0:ab12:4843:cd96:627b:6a04/128]} v4=false v6=false}
Dec 01 21:50:30 eru tailscaled[30516]: LinkChange: all links down; pausing: interfaces.State{defaultRoute= ifs={tailscale0:[100.123.106.4/32 fd7a:115c:a1e0:ab12:4843:cd96:627b:6a04/128]} v4=false v6=false}
Dec 01 21:50:30 eru tailscaled[30516]: [RATELIMIT] format("LinkChange: all links down; pausing: %v")
Dec 01 21:50:32 eru tailscaled[30516]: logtail: internet back up
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.