Azure / Azure/AKS

[BUG] eraser race condition

Open
#5,580 9 comments 0 reactions 2 assignees Claimed by @kaarthis View on GitHub
action-required addon/eraser bug
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe the bug**
The AKS-maintained eraser controller manager has a race condition in the cleanup ImageJob jobs it creates upon startup. When the eraser controller manager starts (or restarts), it creates ImageJob cleanup pods for all currently visible nodes. However, during OS upgrade operations, some nodes are temporarily shut down and not visible to the controller at that moment.

This results in nodes that were being upgraded not receiving their cleanup ImageJob pods, leaving them without image garbage collection until the next scheduled run.

**To Reproduce**
1. Have a multi-node AKS cluster with the eraser controller manager enabled
2. Trigger an OS upgrade on the node pools (e.g., via scheduled maintenance or manual upgrade)
3. During the rolling upgrade, the eraser controller manager either starts fresh or restarts
4. Observe that nodes which were down during the controller's startup don't get eraser pods scheduled

**k8s trace**
Nodes in cluster (13 nodes total):
```
creationTimestamp: "2026-01-28T08:07:44Z" - aks-system-31314909-vmss000000
creationTimestamp: "2026-01-28T08:17:56Z" - aks-worker1-37492165-vmss000000
creationTimestamp: "2026-01-28T08:28:40Z" - aks-infra1-26233419-vmss000005
creationTimestamp: "2026-01-28T08:35:12Z" - aks-infra1-26233419-vmss00000p
creationTimestamp: "2026-01-28T08:41:14Z" - aks-infra1-26233419-vmss00000r
creationTimestamp: "2026-01-28T08:53:00Z" - aks-worker2-76022974-vmss00000b
creationTimestamp: "2026-01-28T09:02:56Z" - aks-system2-28659428-vmss000001
creationTimestamp: "2026-01-28T09:13:37Z" - aks-infra2-30024969-vmss000000
creationTimestamp: "2026-01-28T09:25:28Z" - aks-worker3-37828475-vmss000000
creationTimestamp: "2026-01-28T09:34:25Z" - aks-system-31314909-vmss00000v
creationTimestamp: "2026-01-28T09:38:03Z" - aks-system3-79894051-vmss000000
creationTimestamp: "2026-01-28T09:48:32Z" - aks-infra3-36226154-vmss000000
```

Eraser pods (only 6 nodes have eraser cleanup pods):
```
eraser-aks-infra1-26233419-vmss000005 Completed
eraser-aks-infra1-26233419-vmss00000p Completed
eraser-aks-infra1-26233419-vmss00000r Completed
eraser-aks-system-31314909-vmss000000 Completed
eraser-aks-worker1-37492165-vmss000000 Completed
eraser-aks-worker2-76022974-vmss00000b Completed
eraser-controller-manager Running
```

Missing eraser pods for 6 nodes (all created after ~09:00, during the OS upgrade window):
- aks-system2-28659428-vmss000001
- aks-infra2-30024969-vmss000001
- aks-worker3-37828475-vmss000000
- aks-system-31314909-vmss00000v
- aks-system3-79894051-vmss000000
- aks-infra3-36226154-vmss000000

**Expected behavior**
The eraser cleanup should run on all nodes, including those that were temporarily unavailable during OS upgrades.

**Suggested fix**
Maybe since AKS controls both the node OS upgrades and the eraser deployment, it could coordinate these operations i.e. AKS knows when maintenance windows start and end, and could:
1. Restart or redeploy the eraser controller manager after the maintenance window completes, ensuring all upgraded nodes are visible and receive cleanup jobs, OR
2. Trigger an eraser reconciliation/re-scan after the OS upgrade operation finishes

The upstream eraser project does not currently have a feature to watch for new nodes or reconcile late-joining nodes (see [eraser customization docs](https://eraser-dev.github.io/eraser/docs/customization)). The only scheduling options available are `repeatInterval` (default 24h) and `beginImmediately`, neither of which address this race condition. Since AKS has visibility into both the upgrade lifecycle and the eraser deployment, this coordination would be best handled at the AKS level.

**Environment**
- Kubernetes version 1.32.9
- Eraser version: v1.4.0-7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.