[EKS] [bug]: Incompatible kube-proxy version on upgrading EKS
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
EKS upgrade guide says:
> 5. Update the VPC CNI, CoreDNS, and kube-proxy add-ons.
>
> snip...
>
> 8. After your cluster update is complete, update your nodes to the same Kubernetes version of your updated cluster.
>
> https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html
Between steps 5. and 8., the running kube-proxy can be newer than kubelet on the node. In the case of upgrading Kubernetes 1.18 to 1.19, kube-proxy v1.19 and kubelet v1.18 run on the node at a moment.
Kubernetes official document describes version skew policy as the following:
> * `kube-proxy` must be the same minor version as `kubelet` on the node.
>
> https://kubernetes.io/releases/version-skew-policy/
This version incompatible issue is inevitable when the kube-proxy are deployed by a Kubernetes DaemonSet.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
In practice, I have not seen this break a cluster with an incompatible version. I guess the version skew could be avoidable by the following steps:
1. Expose a version of running kubelet on node labels.
2. Deploy two versions of kube-proxy such as `daemonset/kube-proxy-v1.18` and `daemonset/kube-proxy-v1.19`.
Each daemonset has a `nodeSelector` to select a node with a kubelet which have the same version as kube-proxy.
**Are you currently working around this issue?**
**Additional context**
- Updating a cluster - Amazon EKS
https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html
- Version Skew Policy | Kubernetes
https://kubernetes.io/releases/version-skew-policy/
**Attachments**
nothing.
Contributor guide
Research direction
Start with the EKS upgrade guide and Kubernetes version skew policy linked in the issue. No repository files or tests are named; identify the relevant EKS kube-proxy deployment and node-upgrade entry points before assessing the proposed approach. Done means kube-proxy remains compatible with the kubelet version during upgrades.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100