aws / aws/containers-roadmap

[EKS] [bug] Removed kube-apiservers return 401 Unauthorized instead of closing connection

Open
#1,810 2 comments 19 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

Around the time of the release of EKS 1.23 we started noticing that EKS is more aggressively scaling out/in its kube-apiservers. We are seeing them being replaced more frequently. What we also notice is that whenever a kube-apiserver is removed (it no longer appears in `kubectl get endpoints kubernetes -n default`) it doesn't close existing connections. Instead, whenever a client makes a request to this removed kube-apiserver over an existing connection, the kube-apiserver returns a 401 Unauthorized. This seems to happen every time a kube-apiserver is scaled down. Applications might not be triggered by this 401 Unauthorized to re-establish their connection to the kube-apiserver. Instead, they might think that certain API resources are not available and act accordingly. This happens for example with the latest release of cilium.

I believe that whenever a kube-apiserver is removed as an endpoint, it should also immediately close all of its client connections; forcing the clients to establish a new connection.

Related issue: https://github.com/cilium/cilium/issues/20915

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the behavior described with `kubectl get endpoints kubernetes -n default` while observing existing client connections to the kube-apiserver. Review the related Cilium issue and compare removed-endpoint behavior with the expected immediate connection closure; done means clients are forced to reconnect rather than receiving 401 Unauthorized responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.