Azure / Azure/AKS

[BUG] LocalDNS unit left dead after unclean restart (orphan coredns in cgroup) -> node-local DNS blackhole

Open
#5,930 5 comments 0 reactions 2 assignees Claimed by @saewoni View on GitHub
bug fixing localDNS SR-Support Request
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**What happened:**

With LocalDNS `mode: Preferred` on a System pool, `localdns.service` intermittently fails to restart: a prior run terminates uncleanly and leaves an orphan `coredns` in its delegated cgroup, systemd can't kill the cgroup, restarts trip the StartLimit, and the unit is **left dead**. Pods pointed at `169.254.10.11` then get no DNS and no fallback until a manual `systemctl restart localdns`.

```
localdns.service: Found left-over process (coredns) in control group while starting unit. Ignoring.
localdns.service: This usually indicates unclean termination of a previous run...
localdns.service: Failed to kill control group /localdns.slice/localdns.service, ignoring: Invalid argument
localdns.service: Start request repeated too quickly.
Failed to start localdns.service
```

Impact: ~16 min DNS blackhole per node (e.g. dead `17:47:12 -> 18:03:41 UTC`); **7 of 8** pool nodes hit it within 48h, right after an automatic patch upgrade reimaged the pool.

**Expected:** localdns should clean up the previous run's cgroup before restart (or not StartLimit itself into a dead state); a dead local resolver shouldn't blackhole DNS.

**Key point:** reproduces on the **latest** node image. A node-image upgrade does **not** fix it.

**Environment:**
- K8s `1.36.3`; node image `AKSUbuntu-2404gen2containerd-202608.06.1` (Ubuntu 24.04, kernel `6.8.0-1065-azure`)
- Azure CNI; System pool; LocalDNS `Preferred`/`Enabled` (AKS default, not user-set); West Europe

**Ruled out:** CoreDNS healthy (3/3); conntrack ~720/131072; subnet ~62% used. Not IP/conntrack starvation.

**Mitigation:** setting `localDNSProfile.mode = Disabled` on the pool removes the failure; pools without LocalDNS are unaffected. Requesting a fix to the localdns cgroup teardown + restart handling.

I've also raised this via Azure support (case 2607310050001328) for tracking. Happy to link the two so the investigation stays in sync, and to validate a fixed node image whenever one is ready. Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.