aws / aws/eks-anywhere

Upgrade K8s version from 1.30 to 1.31 stuck on control plane node deployment

Open
#9,819 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**: After the first control plane appeared in k8s, all DaemonSets on this control plane node became stuck with Error: services have not yet been read at least once, cannot construct envvars in events

**What you expected to happen**: All DaemonSets should run without error and the upgrade should complete successfully

**How to reproduce it (as minimally and precisely as possible)**: Upgrade a vSphere k8s cluster from 1.30 to 1.31. The problem may or may not happen, depending on which node the Kube API server VIP resides on at that moment. If it resides on a node with the new k8s version, the problem will not appear.

**Anything else we need to know?**: The problem is described here: https://github.com/kubernetes/kubernetes/issues/127316
Changing the API server IP manually in the file /etc/kubernetes/kubelet/kubeconfig to the node’s IP and then restarting the kubelet daemon fixed the problem

**Environment**:
- EKS Anywhere Release: 0.22.5
- EKS Distro Release: upgrade from v1.30.11-kubernetes-1-30-eks-31 to `v1.31.7-kubernetes-1-31-eks-20`

Contributor guide

Open the contributing guide

Research direction

Reproduce a vSphere cluster upgrade from Kubernetes 1.30 to 1.31 and observe DaemonSets on the first control-plane node. Start with /etc/kubernetes/kubelet/kubeconfig and the related Kubernetes issue #127316, comparing behavior when the API server VIP is on an old versus new-version node. Done means the upgrade completes and all DaemonSets run without the envvar error, without manually changing the kubeconfig.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.