TCP long-lived connections cause service disruption during Kubernetes/Node upgrades in AKS

TCP Long-Lived Connection Hangs During Node Drains/Upgrades causing service disruption:
During AKS cluster or node upgrades, we're experiencing service disruptions due to long-lived TCP connections hanging. This occurs when the Azure Load Balancer (ALB), fronting our Istio Ingress Gateway (deployed as a LoadBalancer service), forwards traffic through an intermediate node.

Our Istio Ingress Gateway uses externalTrafficPolicy: Cluster, meaning the ALB distributes incoming client connections to any node in the cluster. If the target Istio pod is on a different node, the connection is effectively routed through the initial node selected by the ALB.

The problem arises during upgrades: when an intermediate node is cordoned and drained, the ALB fails to recognize this state change. It continues to send (or maintain) TCP connections to this node even as it's being prepared for termination. This leads to connections hanging when the node eventually gets deleted, causing service disruption until clients perform a TCP reset and reconnect to a healthy node. Our observations confirm that nodes continue to accept new connections even after being cordoned/drained.


Note: Traffic comes in through APIM, and that it is APIM that holds the connections open.

We seek guidance on mitigating this issue, specifically how to ensure the ALB promptly stops forwarding new and existing connections to nodes undergoing draining/termination to prevent service interruptions.