systemd-resolved silently drops its 127.0.0.53:53 stub-listener fd from epoll, wedging all libc-path DNS while the daemon stays "alive"

# systemd-resolved silently drops its 127.0.0.53:53 stub-listener fd from epoll, wedging all libc-path DNS while the daemon stays "alive"

## tl;dr

On Bottlerocket aws-k8s-1.34 (observed on 1.60.0 and 1.61.0), systemd-resolved occasionally enters a state where the file descriptor backing its `127.0.0.53:53` UDP stub-listener socket is silently missing from its epoll set. The process keeps running, processes timerfd events normally (~1 wakeup/sec → cache TTL housekeeping is fine), but never recv-loops on the wedged socket. The kernel queues incoming DNS queries until the socket buffer overflows, then drops them. From the libc side this presents as `dial tcp: lookup on 127.0.0.53:53: ... i/o timeout` on every consumer (containerd image pulls, `metricdog` HTTPS posts, any host process using glibc resolution). Pod-network DNS via the cluster's in-cluster resolver is unaffected, so kubelet `NodeReady` conditions stay green and the failure is invisible to standard health checks.

**Every wedged node we've observed has had a public IPv4 address attached.** Same-cluster, same-age, same-Bottlerocket-version private-IP nodes have not been observed to wedge despite carrying comparable host-side DNS load.

We've now hit this several times in ~30 days. We captured the live kernel state of a wedged daemon for the first time, and identified the divergence from a healthy peer.

## Environment

- Bottlerocket: 1.61.0 (aws-k8s-1.34) — also reproduced on 1.60.0 (same variant)
- Kernel: 6.12.83
- containerd: 2.1.7+bottlerocket
- systemd: 252-55.el9_7.7 (per `resolvectl --version`)
- AWS EC2 (mixed x86_64 instance shapes), Karpenter-provisioned
- Cilium CNI; pod-network DNS goes through cluster DNS, host-network DNS through systemd-resolved on 127.0.0.53
- **Every wedged node has had a public IPv4 attached** (see tl;dr note above)

## Symptom

```
dial tcp: lookup on 127.0.0.53:53: read udp 127.0.0.1:-> 127.0.0.53:53: i/o timeout
```

Reproduced on multiple independent nodes across May–June 2026. Bottlerocket version varies (1.60.0 and 1.61.0). On every wedged node, journald shows systemd-resolved logging for the first ~10 seconds after boot then going silent. The last log line is consistently the routine `"Clock change detected. Flushing caches."` emitted around `T+10s` (presumably triggered by chrony stepping the clock during early boot). After that, nothing — for the entire lifetime of the node.

## Live evidence (most recent capture)

Process state on a wedged daemon, captured live via a privileged debug pod (`hostPID: true`):

### Process state
```
State: S (sleeping)
Threads: 1
VmRSS: 9928 kB ← 2.5× a healthy peer
voluntary_ctxt_switches: 34041 ← ~1/sec — timerfds firing normally
nonvoluntary_ctxt_switches: 8892
```

### Kernel stack
```
/proc//wchan: ep_poll

/proc//stack:
[<0>] ep_poll+0x34a/0x3a0
[<0>] do_epoll_wait+0xc1/0xe0
[<0>] __x64_sys_epoll_wait+0x5d/0xf0
[<0>] do_syscall_64+0x79/0x150
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
```

### Syscall args (single thread)
```
/proc//syscall: 232 0x4 0x55f7958fb050 0x2c 0xffffffffffffffff ...
↑ ↑ ↑ ↑
nr epfd maxevents=44 timeout=-1 (infinite)
```

### Wedged listener socket
```
/proc/net/udp row for 127.0.0.53:53:
sl local_address rem_address st tx_queue rx_queue ... uid timeout inode ... drops
5296: 3500007F:0035 00000000:0000 07 00000000:00000000 00000000:00034100 ... 980 0 6159 ... 3840
↑ ↑
213 KB unread 3,840 kernel drops
```

### Fd → socket inode (4 listener sockets: IPv4/IPv6 × 127.0.0.53/.54)
```
readlink /proc//fd/15 → socket:[6159] (127.0.0.53:53, the wedged one)
readlink /proc//fd/16 → socket:[6160] (IPv6 sibling)
readlink /proc//fd/17 → socket:[6161] (127.0.0.54:53, internal DNSSEC validator)
readlink /proc//fd/18 → socket:[6162] (IPv6 sibling)
```

### Epoll set membership — the smoking gun
```
$ cat /proc//fdinfo/4 (the epoll instance)

tfd: 5 events: 19 ... ino:29 sdev:10 (timerfd)
tfd: 6 events: 19 ... ino:29 sdev:10 (timerfd)
tfd: 7 events: 18 ... ino:aae sdev:30
tfd: 8 events: 19 ... ino:29 sdev:10
tfd: 9 events: 19 ... ino:ab1 sdev:9
tfd: 10 events: 19 ... ino:29 sdev:10
tfd: 12 events: 19 ... ino:ab3 sdev:9
tfd: 13 events: 1a ... ino:935 sdev:1b (signalfd)
tfd: 16 events: 19 ... ino:1810 sdev:9 ← UDP listener fd 16
tfd: 17 events: 19 ... ino:1811 sdev:9 ← UDP listener fd 17
tfd: 18 events: 19 ... ino:1812 sdev:9 ← UDP listener fd 18
tfd: 19 events: 19 ... ino:ab9 sdev:9
tfd: 20 events: 19 ... ino:aba sdev:9
tfd: 21 events: 19 ... ino:29 sdev:10
tfd: 22 events: 19 ... ino:29 sdev:10

15 fds total — fd 15 (→ socket:[6159] → 127.0.0.53:53) is conspicuously absent.
```

The other three listener sockets (fds 16, 17, 18) are present. The kernel-level architecture is identical; just one EPOLL_CTL_ADD seems to have been missed (or unregistered without re-add).

## Side-by-side with a healthy peer

We captured the same data from a sibling node: identical Bottlerocket build, identical age, identical instance class, both with public IPv4.

| Field | **Healthy** | **Wedged** |
|---|---|---|
| Bottlerocket | 1.61.0 (aws-k8s-1.34) | 1.61.0 (aws-k8s-1.34) |
| Node age | identical | identical |
| Process state | S (sleeping) | S (sleeping) |
| Threads | 1 | 1 |
| wchan | ep_poll | ep_poll |
| Stack | `ep_poll → do_epoll_wait → __x64_sys_epoll_wait` | identical |
| Syscall | `epoll_wait(4, …, 44, -1)` | identical |
| voluntary_ctxt_switches | 35,187 | 34,041 |
| **VmRSS** | **3,860 kB** | **9,928 kB** |
| **Epoll set size** | **16 fds** | **15 fds** |
| **fd 15 (→ 127.0.0.53:53)** | **in epoll set ✓** | **NOT in epoll set ✗** |
| fd 16/17/18 (sibling listeners) | in epoll set | in epoll set |
| `/proc/net/udp` drops on 127.0.0.53:53 | 0 | 4,160 (at capture) |
| `/proc/net/udp` rx_queue on 127.0.0.53:53 | 0 | 213 KB |
| `resolvectl statistics` | 16,097 transactions, **0 failures** | (daemon never sees queries, so unmeasurable from inside) |

Across every observable field except the missing epoll registration and its downstream consequences (queue backlog, RSS bloat from buffered state) the two processes are byte-for-byte equivalent.

## Hypothesized bug class

The wedge correlates 1:1 with the `"Clock change detected. Flushing caches."` log line at `T+10s`. That message originates from systemd-resolved's clock-step handler, which flushes caches and (we suspect) closes & reopens listener sockets to clear in-flight resolver state. The most plausible bug:

- The clock-step handler closes the old `127.0.0.53:53` socket fd (which auto-removes it from epoll on close), opens a new one, but **misses or races on the `epoll_ctl(EPOLL_CTL_ADD, new_fd, EPOLLIN)`** for the new socket. The sibling listeners go through the same code path successfully — so it's a race / off-by-one in one branch rather than a complete missing call.

We don't have a source-level diagnosis. Open questions for maintainers:
1. Does the cache-flush path indeed touch listener-socket FDs, or are we misreading the trigger?
2. Is the clock-step handler in systemd-resolved a known hot spot for this?
3. Has anyone seen this pattern in containerized/cloud-VM environments where chrony does an initial clock step during boot? Particularly on instances with public IPv4 attached — could the extra `systemd-networkd` configuration around the public interface delay/reorder the resolved startup in a way that exposes the race?

## Trigger / reproducer status

Not deterministic. Empirically:
- A small fraction of newly-provisioned Bottlerocket nodes hit this within seconds of boot
- Multiple same-age, same-OS-version Bottlerocket 1.61 nodes side-by-side: a subset wedged, the rest healthy
- Wedge persists for the entire lifetime of the node — no spontaneous recovery in any observed case
- **Every observed wedge is on a node with a public IPv4 attached.** Private-IP nodes with comparable DNS load have not wedged.

The clock-step at `T+10s` is plausibly the trigger (chrony correcting an initial RTC offset during early boot), but we haven't reproduced it on demand.

## Recovery procedure

`SIGTERM` the wedged systemd-resolved PID from a privileged pod with `hostPID: true`. systemd's `Restart=on-failure` immediately starts a fresh instance with a healthy epoll set. Verified end-to-end: old PID exits, new PID spawned within 1 second, `/proc/net/udp` drops counter reset to 0, the previously-missing fd now correctly registered in the new epoll instance, `resolvectl statistics` accumulating transactions, every libc-path DNS consumer immediately functional. Downtime <1s.

## Detection (for anyone else hitting this)

Cheap test, runnable from a `hostPID` pod or via `nsenter --target 1 --net --mount`:
```sh
awk '$2 ~ /:0035$/ && $NF != "0"' /proc/net/udp
```
Non-zero `drops` on `127.0.0.53:53` is the signal. The definitive confirmation is `/proc//fdinfo/` missing the watch entry for the `127.0.0.53:53` socket fd.

## Asks for Bottlerocket maintainers

1. Confirm whether this is a Bottlerocket-layer bug (systemd integration / Bottlerocket-specific resolved config) or strictly upstream systemd, so we know whether to file in parallel against `systemd/systemd`.
2. Pointers on related issues or known patches we may have missed (none turned up in our search of `bottlerocket-os/bottlerocket` and `systemd/systemd`).
3. Guidance on a Bottlerocket-shipped mitigation. Two plausible options:
- Drop-in `WatchdogSec=30s` + `Restart=on-watchdog` for `systemd-resolved.service`. Requires systemd-resolved to actually issue `sd_notify(WATCHDOG=1)` periodically — needs confirmation in the 252.x build shipped.
- Expose a Bottlerocket setting to disable systemd-resolved entirely and fall back to a simple stub (we've seen this requested in #4732 and #3543, but the demand is broader than just our wedge).

Happy to share additional captures, run further forensic instrumentation on the next occurrence, or test patches against this configuration.