[Feature] Node Disruption Policy
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
Certain cluster/node operations, such as OS upgrades channel update, can trigger unpredictable node re-imaging, which adversely affects customer workloads. AKS users seek to avoid unexpected disruptions and desire greater control over node behaviors to ensure workload reliability and system stability.
This feature will introduce a new managed cluster property that allows users to control whether to block or allow operations that trigger node re-imaging and restrict operations within maintenance window.
## How Node Disruption Policy works
Node Disruption Policy is configured at the cluster level through the `nodeDisruptionProfile` property. When you attempt an operation that requires node reimage, AKS checks the current policy setting:
- **Policy evaluation**: AKS evaluates whether the operation is allowed based on the current policy.
- **Maintenance window check** (if applicable): If using `AllowDuringMaintenanceWindow`, AKS verifies whether the current time falls within the configured maintenance window.
- **Operation execution or blocking**: The node-disruptive operation proceeds if allowed or is rejected with an error message if blocked.
## Policy options
Node Disruption Policy supports three policy configurations:
| Policy | Description | Use case |
|--------|-------------|----------|
| `Allow` | Allows operations that require node reimage to proceed at any time. This is the default behavior. | Use when you want to prioritize applying updates quickly and can tolerate workload disruption. |
| `AllowDuringMaintenanceWindow` | Blocks operations that require node reimage unless they occur within the `aksManagedNodeOSUpgradeSchedule` maintenance window. | Use when you want to limit disruptions to specific maintenance windows that align with your operational schedule. |
| `Block` | Blocks all operations that require node reimage. | Use when you need to prevent any node disruptions, such as during critical business periods or high-traffic events. |
> [!NOTE]
> When using `AllowDuringMaintenanceWindow`, you must have an `aksManagedNodeOSUpgradeSchedule` maintenance window configured. For more information on setting up maintenance windows, see [Use planned maintenance to schedule and control upgrades for your Azure Kubernetes Service cluster][planned-maintenance]. If not, the node-disruptive operations will be allowed.
## Operations covered by Node Disruption Policy
### Cluster-level operations
#### [Network policy][concepts-network] enablement and [Azure CNI Overlay][azure-cni-overlay] upgrade
Nodes need to be reimaged to install the required networking components and configure network rules that secure and manage pod-to-pod communication.
The following table outlines **network policy upgrades that trigger reimage**:
| From | To |
|----|---|
| None (no network policy) | Azure Network Policy |
| None (no network policy) | Calico |
| Azure CNI | Azure CNI Overlay |
| Azure Network Policy | None (no network policy) |
| Calico | None (no network policy) |
> [!NOTE]
> Changing between Azure and Calico network policies after one is already enabled doesn't require a reimage.
#### [Node OS upgrade channel][auto-upgrade-node-os] changes
Each channel uses a different OS patching infrastructure and configuration that you **can't modify** on running nodes.
The following table outlines **node OS upgrade channel changes that trigger reimage**:
| From | To |
|----|---|
| Unmanaged | None |
| Unspecified | Unmanaged |
| SecurityPatch | Unmanaged |
| NodeImage | Unmanaged |
| None | Unmanaged |
| Unspecified | Unmanaged |
| Unmanaged | SecurityPatch |
| Unmanaged | NodeImage |
#### [IPv6 dual-stack][configure-dual-stack] enablement
Nodes need both IPv4 and IPv6 IP configurations and networking stack updates (such as nftables rules) to support dual-stack communication.
The following table outlines **IP configuration and networking stack updates that trigger reimage**:
| From | To |
|----|---|
| IPv4 only | IPv4 + IPv6 (dual-stack) |
#### [Cilium data plane][azure-cni-cilium] changes
You need to install or remove eBPF programs that handle packet processing at the kernel level.
The following table outlines **Cilium data plane changes that trigger reimage**:
| From | To |
|----|---|
| None | Cilium |
| Cilium | None |
#### [HTTP proxy][http-proxy] configuration updates
All node components (containerd, kubelet, system services) need the updated proxy configuration applied system-wide. When you update HTTP proxy configuration, AKS automatically reimages all node pools in the cluster.
Node Disruption Policy **triggers reimage** when you modify any of the following HTTP proxy configuration properties or perform any of the following operations:
- `httpProxy`: Proxy URL for HTTP connections
- `httpsProxy`: Proxy URL for HTTPS connections
- `noProxy`: List of destinations to exclude from proxying
- `trustedCa`: Base64 encoded alternative CA certificate
- Enable HTTP proxy on a cluster (with `--enable-http-proxy`)
- Disable HTTP proxy on a cluster (with `--disable-http-proxy`)
- Reenable HTTP proxy on a cluster that previously had it disabled
#### [Custom CA certificates][custom-ca] updates
You must install new CA certificates in the OS trust store to affect TLS validation for internal services and private registries.
Node Disruption Policy **triggers reimage** when you add, remove, or update custom CA certificates.
#### [Kubelet identity][concepts-identity] changes
You must apply new identity credentials to the node configuration. This includes initial identity assignment, identity updates, and service principal profile resets.
Node Disruption Policy **triggers reimage** when you update a managed identity or user-assigned managed identity for the kubelet.
#### [Private DNS zone][access-private-cluster] changes
You must update DNS resolver settings to resolve the private API server endpoint using the new DNS zone.
Node Disruption Policy **triggers reimage** when you modify a private DNS zone configuration in a private cluster.
#### [Private cluster][access-private-cluster] enablement
You must reconfigure networking and DNS settings to communicate with the API server through the private endpoint.
Node Disruption Policy **triggers reimage** when you update an existing public cluster to a private cluster with API server VNet integration.
#### [eBPF host routing][advanced-container-networking] changes
You need to install or remove eBPF programs that provide high-performance packet forwarding (BpfVeth acceleration mode).
The following table outlines **eBPF host routing changes that trigger reimage**:
| From | To |
|----|---|
| Standard routing | eBPF host routing enabled |
| eBPF host routing enabled | Standard routing |
#### [TLS bootstrap][certificate-rotation] settings updates
When you update TLS bootstrap settings, you need to generate new bootstrap tokens and apply them to nodes to enable kubelet TLS bootstrapping for certificate rotation.
Node Disruption Policy **triggers reimage** when you update agent pool TLS bootstrap settings.
### Node pool-level operations
These operations affect only the specific node pools where changes are made, triggering a rolling reimage within those node pools:
#### [Local DNS][localdns-custom] profile updates
You need to apply changes to the DNS caching daemon and DNS forwarding rules at the node level.
Node Disruption Policy **triggers reimage** when you modify a LocalDNS profile configuration.
#### [Trusted Launch][use-trusted-launch] security changes
You can't change VM firmware configuration and boot process settings on running VMs . These changes you to recreate the VMs.
The following table outlines **Trusted Launch security changes that trigger reimage**:
| Configuration | From | To |
|--------------|------|-----|
| vTPM (virtual Trusted Platform Module) | Disabled | Enabled |
| vTPM (virtual Trusted Platform Module) | Enabled | Disabled |
| Secure Boot | Disabled | Enabled |
| Secure Boot | Enabled | Disabled |
#### [Artifact streaming][artifact-streaming] changes
You need to install or remove artifact streaming components to enable faster container image pulling by streaming image layers on-demand.
The following table outlines **artifact streaming changes that trigger reimage**:
| From | To |
|----|---|
| Disabled | Enabled |
| Enabled | Disabled |
#### [Windows GMSA][use-gmsa] profile updates (Windows node pools only)
You need to apply new GMSA settings, DNS server configuration, and domain join credentials for Active Directory integration.
Node Disruption Policy **triggers reimage** when you modify GMSA configurations on Windows node pools
### Operations not yet covered by Node Disruption Policy
The following configuration changes require node reimage but **aren't yet covered by Node Disruption Policy**. They will be covered in a future Kubernetes minor version updates as this will be a behavior change.
After making these configuration changes, you must manually run [`az aks nodepool upgrade`][az-aks-nodepool-upgrade] with `--node-image-only` to apply the changes to your nodes.
- **[SSH configuration][manage-ssh-access] changes**: Changing SSH access methods (Disabled SSH, Entra ID based SSH, or Local User SSH) or updating SSH public keys on node pools.
- **[IMDS restriction][imds-restriction] changes**: Enabling or disabling Instance Metadata Service (IMDS) restriction to block pod access to the IMDS endpoint.
- **Bootstrap profile ACR name changes**: Changing the Azure Container Registry (ACR) used for bootstrap profiles in [network isolated clusters][network-isolated].
- **[Outbound type][egress-outboundtype] changes**: Modifying the cluster's outbound connectivity type (loadBalancer, userDefinedRouting, managedNATGateway, or userAssignedNATGateway).
### Upgrade operations not controlled by Node Disruption Policy
Node Disruption Policy **doesn't control** the following upgrade operations. These upgrade operations proceed regardless of your policy setting. Upgrades are either customer-initiated or initiated by AKS within planned maintenance windows and are intentionally kept out of Node Disruption Policy scope to allow them to proceed as scheduled.
- **[Node image version][auto-upgrade-node-os] updates**: Upgrading to a new node OS image version (manually or through automatic upgrade channels). This is the most common reimage operation and includes security patches, OS updates, and AKS node image releases.
- **[Kubernetes version][auto-upgrade-cluster] upgrades**: Upgrading the Kubernetes version on a node pool, which applies new Kubernetes binaries, updated kubelet configuration, and OS-level changes.
### Recovery operations not controlled by Node Disruption Policy
Node Disruption Policy **doesn't control** the following automated recovery operations. These operations can occur regardless of your policy setting to ensure cluster health and recovery.
- **Node pool configuration rollback**: When a node pool update operation fails due to invalid configuration or infrastructure issues, AKS automatically rolls back to the last known good state and reimages nodes to revert configuration.
- **Admin cluster restore operations**: When Azure support engineers perform administrative cluster restoration during incident resolution, nodes are reimaged to ensure consistency between control plane state and node configuration.
- **Node identity credential updates**: AKS periodically updates node identity credentials for security and compliance. These system-initiated updates trigger node reimages to apply the new credentials across all node pools.
Contributor guide
Assessment
This issue has not been assessed yet.