aws / aws/containers-roadmap

[EKS] [Managed NodeGroup upgrade] EKS doesn't deregister a node from load balancer before draining pods

Open
#2,748 1 comment 0 reactions 0 assignees View on GitHub
EKS EKS Managed Nodes
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

The documentation located at https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html states that during the "Scale up phase" :
"It marks nodes as unschedulable to avoid scheduling new Pods. It also **labels nodes with node.kubernetes.io/exclude-from-external-load-balancers=true to remove the old nodes from load balancers before terminating the nodes**."

During a managed nodegroup upgrade, we witnessed that it was not the case :

instead of adding the label to the node for the loadbalancer controller to remove the corresponding targetgroup from the loadbalancer just after cordoning the node, we witnessed that it only adds the label after draining the pods from the kubernetes node so the node is staying behind the loadbalancer whereas pods are being drained from the node which leads to downtime.

Context:
- EKS cluster 1.31 with a managed nodegroup
- AWS LoadBalancer controller 2.10
- Traefik ingress controller with a k8s service configured with `externalTrafficPolicy: Local`
- an echoserver application (deployment + service) exposed via a traefik "IngressRoute"

Issue detection :
During upgrade EKS from 1.31 to 1.32, multiple requests failed

*** Actual Order of Operations:
- Add new 1.32 nodes
- Cordon old 1.31 nodes (mark unschedulable)
- Pick first old node and drain pods
- Add the exclude-from-external-load-balancers label (somewhere in this process)
- Terminate node
(repeat with other old nodes)

*** Expected Order (to avoid downtime):
- Add new 1.32 nodes
- Cordon old 1.31 nodes (mark unschedulable)
- Pick first old node and adds the exclude-from-external-load-balancers label (remove from load balancer) **as written in the documentation**
- Deregistration API call made
- Wait for load balancer to de-register ("draining" in-flight connections, maximum being the "Deregistration delay")
- Drain pods
- Terminate node
(repeat with other old nodes)

During the drain phase, an old node is:

- Still registered in the load balancer target groups
- Having its pods evicted/terminated
- Receiving live traffic from the load balancer (until the unhealthy checks are done and the node becomes unhealthy)

This causes connection failures and 5xx errors because the load balancer sends traffic to a node that's actively draining its workloads.

With `externalTrafficPolicy: Cluster` : Some requests might route to pods on other nodes (bad, but survivable)
With `externalTrafficPolicy: Local` : The moment the termination process begins, that port is closed to external traffic, meaning there is no grace period for the load balancer to remove the node from its backend pool, and any traffic sent to the node is silently lost

Contributor guide

Open the contributing guide

Research direction

Start with the linked EKS managed node update behavior documentation and the reproduction context: EKS 1.31/1.32, AWS LoadBalancer Controller 2.10, Traefik, and externalTrafficPolicy: Local. Compare the observed and expected operation orders; done means confirming whether deregistration precedes pod draining or documenting the service limitation and required behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.