kubernetes-sigs / kubernetes-sigs/cluster-api

Kubernetes version skew check for machine pools

Open
#12,732 9 comments 0 reactions 0 assignees View on GitHub
area/machinepool kind/feature lifecycle/stale needs-triage priority/important-longterm
Dominant language
Go
Stars
4.3k
Forks
1.6k
Avg merge
1d 3h
Merged PRs (30d)
113

Description

### What would you like to be added (User Story)?

The [version skew policy](https://kubernetes.io/releases/version-skew-policy/) should be respected when rolling out nodes of a machine pool. For example, if the control plane nodes aren't up to date yet, workers with a newer version must not be created.

### Detailed Description

The implementation in infra providers can vary quite a lot, so this issue is to try and find a common ground. For instance, contract/API fields, a helper function or documented implementation recommendation may help here.

Example:

- Before: KCP on Kubernetes version v1.29.0, MachinePool on v1.29.0
- User applies v1.30.0 on both objects at once
- If the machine pool controller doesn't check for version skew, or still sees the old MachinePool object's `spec.version` (= v1.29.0) in its version skew check, it might already roll out workers with v1.30.0 which can lead to an outage if these versions aren't compatible (which the policy doesn't guarantee at all: "kubelet must not be newer than kube-apiserver").

At time of writing (2025-09-04), there was no known MachinePool controller implementation with code for version skew check. Initial attempts have been made in infra providers: in [this CAPA PR](https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5560), we found that the CAPA controller might see an old `MachinePool` object, which happens because `kubectl apply` or `helm {install,upgrade}`, for example, deploy objects in mostly random order and without time guarantees. There's no good way to find out whether `MachinePool.spec.version` and the Kubernetes version implicitly defined by `AWSMachinePool.spec.awsLaunchTemplate.ami.id` (the VM image) are in sync. Only a new field/label on the `MachinePool` would help in that case.

If the infra provider pauses reconciliation of the machine pool cloud resources while the control plane is outdated/skewed, it may, depending on implementation, still need to ensure that the bootstrap token for newly-created workers gets refreshed (e.g. AWS: EC2 user data in launch template). This can be tricky to implement and test.

### Anything else you would like to add?

_No response_

### Label(s) to be applied

/kind feature
/area machinepool

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the MachinePool controller implementations and the version skew policy, then examine the CAPA PR #5560 for the stale-object and bootstrap-token concerns described here. A completed solution should establish a common contract or implementation recommendation that prevents incompatible worker rollouts while handling control-plane updates and refreshed bootstrap data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend-api-design, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.