Azure / Azure/AKS

[Feature] Pod Startup Throttling After Reboot / Thundering Herd Issue

Open
#5,263 3 comments 0 reactions 1 assignee Assigned to @AllenWen-at-Azure View on GitHub
feature-request Scale and Performance stale
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
If a Windows Server 2022 AKS node reboots, the restarting of dozens of pods simultaneously, and the overhead of setting up the pod sandbox, networking, etc. often causes kubelet to run slowly. This is enough for the node to flap between Ready and NotReady as health checks time out.

The `aks-auto-repair` feature in the control plane then may repeatedly reboot the node, causing significant disruption.

**Describe the solution you'd like**
Given that often it is the creation of the sandbox container, which times out, the most logical solution is to add some sort of throttling or rate limiting, before this step. The best place for this seems to be the execution of CNI plugins, which can be a good opportunity to throttle pod startup, without creating much load on the underlying system.

**Describe alternatives you've considered**
There are CNI solutions, such as pod-pacemaker, which is designed to run on Linux Kubernetes nodes. This works by creating effectively a dummy CNI plugin, which simply delays/throttles the pod startup, hooking in early enough in the lifecycle.

However, installing custom CNI plugins isn't supported on Windows nodes, so there's not much point currently in attempting to port this.

Some solutions suggest using init containers with randomised delays, but this doesn't work to solve the issue, as the bulk of the overhead is the creation of the pod sandbox, which would still happen.

**Additional context**
In a default Kubernetes cluster, where `aks-auto-repair` has not been disabled by Azure support, this can cause some difficult to resolve outages that can effectively spread to and impact all Windows nodes on which a pod may be rescheduled. This caused a significant outage in one of our production clusters over the weekend that could only be resolved by manually scaling down deployments to 0, then re-scaling back up in stages.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.