[Feature] Drain-Aware Node Upgrades
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
Azure Kubernetes Service (AKS) is working on a new capability that gives stateful and disruption-sensitive workloads more control over when individual nodes can be upgraded.
# 📋 What problem are we solving?
Some workloads need to evaluate replication health, quorum, replica roles, topology, or application state before a node can be safely drained.
Pod Disruption Budgets provide general availability protection, but they don’t allow an application to decide exactly which node can be disrupted or in what order.
Today, these workloads might experience blocked upgrades, repeated upgrade attempts, retained surge capacity, or require custom node-replacement workflows.
# ✨ What we’re proposing
Customers or their workload automation can mark a node as temporarily unavailable for upgrade:
```bash
kubectl label node kubernetes.azure.com/upgrade-status=paused
```
After a normal AKS node-pool upgrade starts, AKS will:
- Skip paused nodes before cordoning or draining them.
- Continue upgrading eligible nodes.
- Keep paused nodes healthy and schedulable.
- Avoid creating or retaining surge capacity for paused nodes.
- Report paused nodes separately from unexpected upgrade failures.
To release a node:
```bash
kubectl label node kubernetes.azure.com/upgrade-status-
```
No additional pool-level upgrade request is required after releasing the node.
# 🔄 Example upgrade flow
Assume a 100-node pool requires an upgrade and 80 nodes are paused:
- AKS upgrades the 20 eligible nodes.
- Active progress is measured across those 20 nodes.
- The 80 paused nodes remain schedulable and don’t retain surge replacements.
- As the workload releases nodes, AKS upgrades them to the pool’s desired version.
- The pool fully converges after all 100 nodes are upgraded.
# 🏷️ Paused versus quarantined nodes
State | Owner | Meaning | AKS behavior
-- | -- | -- | --
upgrade-status=paused | Customer | The node is healthy but not currently safe to disrupt. | Skip it before execution without cordoning, draining, retaining surge capacity, or reporting a drain error.
upgrade-status=Quarantined | AKS | AKS attempted to upgrade the node, but the drain failed unexpectedly. | Apply the existing undrainableNodeBehavior and maxBlockedNodes behavior.
A separate `Paused` value is necessary because `Quarantined` represents an upgrade failure after AKS has already cordoned and attempted to drain the node. A quarantined node consumes an upgrade attempt, can retain surge capacity, and affects operation status.
A paused node instead represents customer intent before disruption and provides a release signal for later reconciliation.
# 🚨 Break-glass upgrades
For urgent upgrades, such as critical security updates, platform owners could use the existing AKS force-upgrade override.
For example:
```
az aks update \
--resource-group \
--name \
--enable-force-upgrade \
--upgrade-override-until
az aks nodepool upgrade \
--resource-group \
--cluster-name \
--name \
--kubernetes-version
```
While the override is valid, AKS would ignore the pause label for that upgrade and report that the override was used.
# 🎯 Example scenario
## Before ❌
A database operator knows that only one replica is currently safe to disrupt. AKS independently selects another node, its drain fails, and the upgrade retains replacement capacity or requires another attempt.
## After ✅
The workload automation pauses nodes that aren’t ready. AKS upgrades the eligible replica first. When application health allows the next disruption, the automation removes that node’s pause label and AKS continues reconciliation.
# ⚠️ Initial limitations
- Customers or customer-managed automation are responsible for adding and removing the pause label.
- AKS doesn’t initially infer readiness from workload health or metrics.
- The feature changes node eligibility after an operation starts; it doesn’t change how upgrades, automatic upgrades, or Fleet upgrade runs are triggered.
- Unexpected drain failures continue to use existing quarantine behavior.
- AKS will prevent a pool from advancing when doing so would introduce more than two node versions.
- AKS doesn’t support node images older than 90 days, so long-lived pauses remain subject to AKS support and security requirements.
# 🔭 Future direction
We’re also exploring:
- Periodic reconciliation of released nodes using maintenance settings.
- An AKS-managed controller that automatically applies and removes pause labels for selected workloads.
>Note: The label, API, CLI, status, and reconciliation examples illustrate the proposed experience and might change before release.
Contributor guide
Assessment
This issue has not been assessed yet.