Upgrade K8s version from 1.30 to 1.31 stuck on control plane node deployment
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**: After the first control plane appeared in k8s, all DaemonSets on this control plane node became stuck with Error: services have not yet been read at least once, cannot construct envvars in events
**What you expected to happen**: All DaemonSets should run without error and the upgrade should complete successfully
**How to reproduce it (as minimally and precisely as possible)**: Upgrade a vSphere k8s cluster from 1.30 to 1.31. The problem may or may not happen, depending on which node the Kube API server VIP resides on at that moment. If it resides on a node with the new k8s version, the problem will not appear.
**Anything else we need to know?**: The problem is described here: https://github.com/kubernetes/kubernetes/issues/127316
Changing the API server IP manually in the file /etc/kubernetes/kubelet/kubeconfig to the node’s IP and then restarting the kubelet daemon fixed the problem
**Environment**:
- EKS Anywhere Release: 0.22.5
- EKS Distro Release: upgrade from v1.30.11-kubernetes-1-30-eks-31 to `v1.31.7-kubernetes-1-31-eks-20`
Contributor guide
Research direction
Reproduce a vSphere cluster upgrade from Kubernetes 1.30 to 1.31 and observe DaemonSets on the first control-plane node. Start with /etc/kubernetes/kubelet/kubeconfig and the related Kubernetes issue #127316, comparing behavior when the API server VIP is on an old versus new-version node. Done means the upgrade completes and all DaemonSets run without the envvar error, without manually changing the kubeconfig.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100