[Feature] Managed azure-wi-webhook-controller-manager should enforce replica spread across nodes to prevent fail-closed admission failures
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
The AKS-managed Azure Workload Identity mutating admission webhook (`azure-wi-webhook-controller-manager`, deployed in `kube-system`) runs multiple replicas with a nodeAffinity preference for the system node pool, but no `podAntiAffinity` or `topologySpreadConstraints`. Without a spreading constraint, replicas will co-locate on a single node. This design is prone to intermittent admission timeouts that result in pod creation failures for all workload-identity pods during transient node pressure events.
The `MutatingWebhookConfiguration` (`azure-wi-webhook-mutating-webhook-configuration`) is configured with `failurePolicy: Fail`, `timeoutSeconds: 10`, and `objectSelector: azure.workload.identity/use=true`. This means transient network or CPU pressure on that single node will simultaneously degrade all webhook replicas and cause admission failures for every workload-identity pod, with no fallback.
Because this Deployment is fully AKS-managed and reconciled by the managed addon, customers cannot apply scheduling constraints durably. This is confirmed by two labels on the Deployment: `kubernetes.azure.com/managedby: aks` and Flux HelmRelease annotation `helm.toolkit.fluxcd.io/name: workload-identity-adapter-helmrelease`. Any manual edit to replicas, anti-affinity, resources, or timeout is overwritten on the next reconciliation cycle.
**Describe the solution you'd like**
Please add a `topologySpreadConstraints` entry to the managed Deployment spec with `topologyKey: kubernetes.io/hostname` and `maxSkew: 1`, so that replicas are scheduled across different nodes.
Goal: a single node's transient network or CPU pressure can no longer degrade all webhook replicas simultaneously.
**Describe alternatives you've considered**
A `requiredDuringSchedulingIgnoredDuringExecution` pod anti-affinity rule on `kubernetes.io/hostname` would also achieve replica spreading and is an acceptable alternative to topologySpreadConstraints. Either approach resolves the co-location risk.
Manual edits to the Deployment are not a durable solution because the managed addon reconciles them back on the next cycle. There is no supported mechanism for customers to patch AKS-managed Deployments persistently.
**Additional context**
For additional resilience in churn-heavy clusters, the following are also requested for Microsoft's consideration:
1. Raise `timeoutSeconds` from 10 to 30 to provide more tolerance for brief apiserver-to-webhook latency.
2. Increase the replica count from 2 to 3 and raise the per-pod CPU limit above its current 200m (current resource profile: `requests cpu=100m/mem=20Mi`, `limits cpu=200m/mem=300Mi`), so the webhook has more headroom during pod-creation storms.
Contributor guide
Assessment
This issue has not been assessed yet.