aws / aws/containers-roadmap

[EKS] [bug]: Incompatible kube-proxy version on upgrading EKS

Open
#1,379 0 comments 0 reactions 0 assignees View on GitHub
EKS EKS Add-Ons Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

EKS upgrade guide says:
> 5. Update the VPC CNI, CoreDNS, and kube-proxy add-ons.
>
> snip...
>
> 8. After your cluster update is complete, update your nodes to the same Kubernetes version of your updated cluster.
>
> https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

Between steps 5. and 8., the running kube-proxy can be newer than kubelet on the node. In the case of upgrading Kubernetes 1.18 to 1.19, kube-proxy v1.19 and kubelet v1.18 run on the node at a moment.

Kubernetes official document describes version skew policy as the following:

> * `kube-proxy` must be the same minor version as `kubelet` on the node.
>
> https://kubernetes.io/releases/version-skew-policy/

This version incompatible issue is inevitable when the kube-proxy are deployed by a Kubernetes DaemonSet.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

In practice, I have not seen this break a cluster with an incompatible version. I guess the version skew could be avoidable by the following steps:

1. Expose a version of running kubelet on node labels.
2. Deploy two versions of kube-proxy such as `daemonset/kube-proxy-v1.18` and `daemonset/kube-proxy-v1.19`.
Each daemonset has a `nodeSelector` to select a node with a kubelet which have the same version as kube-proxy.

**Are you currently working around this issue?**

**Additional context**

- Updating a cluster - Amazon EKS
https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html
- Version Skew Policy | Kubernetes
https://kubernetes.io/releases/version-skew-policy/

**Attachments**

nothing.

Contributor guide

Open the contributing guide

Research direction

Start with the EKS upgrade guide and Kubernetes version skew policy linked in the issue. No repository files or tests are named; identify the relevant EKS kube-proxy deployment and node-upgrade entry points before assessing the proposed approach. Done means kube-proxy remains compatible with the kubelet version during upgrades.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.