[EKS] [Managed NodeGroup upgrade] EKS doesn't deregister a node from load balancer before draining pods
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
The documentation located at https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html states that during the "Scale up phase" :
"It marks nodes as unschedulable to avoid scheduling new Pods. It also **labels nodes with node.kubernetes.io/exclude-from-external-load-balancers=true to remove the old nodes from load balancers before terminating the nodes**."
During a managed nodegroup upgrade, we witnessed that it was not the case :
instead of adding the label to the node for the loadbalancer controller to remove the corresponding targetgroup from the loadbalancer just after cordoning the node, we witnessed that it only adds the label after draining the pods from the kubernetes node so the node is staying behind the loadbalancer whereas pods are being drained from the node which leads to downtime.
Context:
- EKS cluster 1.31 with a managed nodegroup
- AWS LoadBalancer controller 2.10
- Traefik ingress controller with a k8s service configured with `externalTrafficPolicy: Local`
- an echoserver application (deployment + service) exposed via a traefik "IngressRoute"
Issue detection :
During upgrade EKS from 1.31 to 1.32, multiple requests failed
*** Actual Order of Operations:
- Add new 1.32 nodes
- Cordon old 1.31 nodes (mark unschedulable)
- Pick first old node and drain pods
- Add the exclude-from-external-load-balancers label (somewhere in this process)
- Terminate node
(repeat with other old nodes)
*** Expected Order (to avoid downtime):
- Add new 1.32 nodes
- Cordon old 1.31 nodes (mark unschedulable)
- Pick first old node and adds the exclude-from-external-load-balancers label (remove from load balancer) **as written in the documentation**
- Deregistration API call made
- Wait for load balancer to de-register ("draining" in-flight connections, maximum being the "Deregistration delay")
- Drain pods
- Terminate node
(repeat with other old nodes)
During the drain phase, an old node is:
- Still registered in the load balancer target groups
- Having its pods evicted/terminated
- Receiving live traffic from the load balancer (until the unhealthy checks are done and the node becomes unhealthy)
This causes connection failures and 5xx errors because the load balancer sends traffic to a node that's actively draining its workloads.
With `externalTrafficPolicy: Cluster` : Some requests might route to pods on other nodes (bad, but survivable)
With `externalTrafficPolicy: Local` : The moment the termination process begins, that port is closed to external traffic, meaning there is no grace period for the load balancer to remove the node from its backend pool, and any traffic sent to the node is silently lost
Contributor guide
Research direction
Start with the linked EKS managed node update behavior documentation and the reproduction context: EKS 1.31/1.32, AWS LoadBalancer Controller 2.10, Traefik, and externalTrafficPolicy: Local. Compare the observed and expected operation orders; done means confirming whether deregistration precedes pod draining or documenting the service limitation and required behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100