kubernetes-sigs / kubernetes-sigs/cluster-api

MachineDeployment observedGeneration updated before other status fields

Open
#13,738 11 comments 0 reactions 1 assignee Claimed by @mattsu2020 View on GitHub
kind/bug needs-priority needs-triage
Dominant language
Go
Stars
4.3k
Forks
1.6k
Avg merge
1d 3h
Merged PRs (30d)
113

Description

### What steps did you take and what happened?

Similar to https://github.com/kubernetes-sigs/cluster-api/issues/10059 but for MachineDeployments.
I encountered an inconsistency (even if it does not last for long): I use a k8s operator to manage CAPI resources, and at some moment it changed only `bootstrap.configRef` (not k8sVersion) from a MachineDeployment. The operator checks MD status to determine if it's ready and it can proceed with further steps, but there is a time window in which the ObservedGeneration (from status and all conditions) is updated to the last one without even having started to roll out. If we look at the MachineDeployment at that specific point in time, we would think the latest generation is fully ready when it has not even re-rolled one machine.
A few iterations later we can see the `RollingOut` condition changed to `True` and `upToDateReplicas == 0`.
Here is a condensed view of the MD transitions captured with `kubectl get machinedeployments --watch -oyaml`

1. Object is updated and CAPI did not react

```yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
annotations:
machinedeployment.clusters.x-k8s.io/revision: "2"
creationTimestamp: "2026-05-22T13:39:10Z"
finalizers:
- cluster.x-k8s.io/machinedeployment
generation: 3
labels:
cluster.x-k8s.io/cluster-name: workload-docker-01
name: workload-docker-01-wave-1
namespace: c-workload-docker-01
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
name: workload-docker-01
uid: d07ab90f-018e-4ded-adc7-696079357b56
resourceVersion: "7888632"
uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2026-05-22T13:39:57Z"
message: ""
observedGeneration: 2
reason: Available
status: "True"
type: Available
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 2
reason: NotRollingOut
status: "False"
type: RollingOut
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 2
reason: NotRemediating
status: "False"
type: Remediating
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 2
reason: NotScalingDown
status: "False"
type: ScalingDown
- lastTransitionTime: "2026-05-22T13:39:16Z"
message: ""
observedGeneration: 2
reason: NotScalingUp
status: "False"
type: ScalingUp
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 2
reason: Ready
status: "True"
type: MachinesReady
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 2
reason: UpToDate
status: "True"
type: MachinesUpToDate
- lastTransitionTime: "2026-05-22T13:39:11Z"
message: ""
observedGeneration: 2
reason: NotPaused
status: "False"
type: Paused
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 2
reason: NotDeleting
status: "False"
type: Deleting
observedGeneration: 2
phase: Running
readyReplicas: 1
replicas: 1
selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
upToDateReplicas: 1
```

2. First CAPI reconciliation setting observedGeneration

```yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
annotations:
machinedeployment.clusters.x-k8s.io/revision: "3"
creationTimestamp: "2026-05-22T13:39:10Z"
finalizers:
- cluster.x-k8s.io/machinedeployment
generation: 3
labels:
cluster.x-k8s.io/cluster-name: workload-docker-01
name: workload-docker-01-wave-1
namespace: c-workload-docker-01
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
name: workload-docker-01
uid: d07ab90f-018e-4ded-adc7-696079357b56
resourceVersion: "7888650"
uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2026-05-22T13:39:57Z"
message: ""
observedGeneration: 3
reason: Available
status: "True"
type: Available
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: NotRollingOut
status: "False"
type: RollingOut
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 3
reason: NotRemediating
status: "False"
type: Remediating
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: NotScalingDown
status: "False"
type: ScalingDown
- lastTransitionTime: "2026-05-22T13:39:16Z"
message: ""
observedGeneration: 3
reason: NotScalingUp
status: "False"
type: ScalingUp
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: Ready
status: "True"
type: MachinesReady
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: UpToDate
status: "True"
type: MachinesUpToDate
- lastTransitionTime: "2026-05-22T13:39:11Z"
message: ""
observedGeneration: 3
reason: NotPaused
status: "False"
type: Paused
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 3
reason: NotDeleting
status: "False"
type: Deleting
observedGeneration: 3
phase: Running
readyReplicas: 1
replicas: 1
selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
upToDateReplicas: 1
```

3. CAPI reporting RollingOut condition for the first time

```yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
annotations:
machinedeployment.clusters.x-k8s.io/revision: "3"
creationTimestamp: "2026-05-22T13:39:10Z"
finalizers:
- cluster.x-k8s.io/machinedeployment
generation: 3
labels:
cluster.x-k8s.io/cluster-name: workload-docker-01
name: workload-docker-01-wave-1
namespace: c-workload-docker-01
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
name: workload-docker-01
uid: d07ab90f-018e-4ded-adc7-696079357b56
resourceVersion: "7888674"
uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2026-05-22T13:39:57Z"
message: ""
observedGeneration: 3
reason: Available
status: "True"
type: Available
- lastTransitionTime: "2026-05-22T15:08:01Z"
message: |-
Rolling out 1 not up-to-date replicas
* KubeadmConfig is not up-to-date
observedGeneration: 3
reason: RollingOut
status: "True"
type: RollingOut
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 3
reason: NotRemediating
status: "False"
type: Remediating
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: NotScalingDown
status: "False"
type: ScalingDown
- lastTransitionTime: "2026-05-22T13:39:16Z"
message: ""
observedGeneration: 3
reason: NotScalingUp
status: "False"
type: ScalingUp
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: Ready
status: "True"
type: MachinesReady
- lastTransitionTime: "2026-05-22T15:08:01Z"
message: |-
* Machine workload-docker-01-wave-1-lldqr-hhcrl:
* KubeadmConfig is not up-to-date
observedGeneration: 3
reason: NotUpToDate
status: "False"
type: MachinesUpToDate
- lastTransitionTime: "2026-05-22T13:39:11Z"
message: ""
observedGeneration: 3
reason: NotPaused
status: "False"
type: Paused
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 3
reason: NotDeleting
status: "False"
type: Deleting
deprecated:
v1beta1:
availableReplicas: 1
conditions:
- lastTransitionTime: "2026-05-22T13:39:56Z"
status: "True"
type: Ready
- lastTransitionTime: "2026-05-22T13:39:56Z"
status: "True"
type: Available
- lastTransitionTime: "2026-05-22T15:08:00Z"
reason: WaitingForMachineSet
severity: Info
status: "False"
type: MachineSetReady
readyReplicas: 1
unavailableReplicas: 1
updatedReplicas: 0
observedGeneration: 3
phase: Running
readyReplicas: 1
replicas: 1
selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
upToDateReplicas: 1
```

4. Finally CAPI reports upToDateReplicas: 0

```yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
annotations:
machinedeployment.clusters.x-k8s.io/revision: "3"
creationTimestamp: "2026-05-22T13:39:10Z"
finalizers:
- cluster.x-k8s.io/machinedeployment
generation: 3
labels:
cluster.x-k8s.io/cluster-name: workload-docker-01
name: workload-docker-01-wave-1
namespace: c-workload-docker-01
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
name: workload-docker-01
uid: d07ab90f-018e-4ded-adc7-696079357b56
resourceVersion: "7888696"
uid: 025e14c3-ad3c-40aa-8ad3-00c411587b23
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2026-05-22T13:39:57Z"
message: ""
observedGeneration: 3
reason: Available
status: "True"
type: Available
- lastTransitionTime: "2026-05-22T15:08:01Z"
message: |-
Rolling out 1 not up-to-date replicas
* KubeadmConfig is not up-to-date
observedGeneration: 3
reason: RollingOut
status: "True"
type: RollingOut
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 3
reason: NotRemediating
status: "False"
type: Remediating
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: NotScalingDown
status: "False"
type: ScalingDown
- lastTransitionTime: "2026-05-22T13:39:16Z"
message: ""
observedGeneration: 3
reason: NotScalingUp
status: "False"
type: ScalingUp
- lastTransitionTime: "2026-05-22T14:39:14Z"
message: ""
observedGeneration: 3
reason: Ready
status: "True"
type: MachinesReady
- lastTransitionTime: "2026-05-22T15:08:01Z"
message: |-
* Machine workload-docker-01-wave-1-lldqr-hhcrl:
* KubeadmConfig is not up-to-date
observedGeneration: 3
reason: NotUpToDate
status: "False"
type: MachinesUpToDate
- lastTransitionTime: "2026-05-22T13:39:11Z"
message: ""
observedGeneration: 3
reason: NotPaused
status: "False"
type: Paused
- lastTransitionTime: "2026-05-22T13:39:12Z"
message: ""
observedGeneration: 3
reason: NotDeleting
status: "False"
type: Deleting
deprecated:
v1beta1:
availableReplicas: 1
conditions:
- lastTransitionTime: "2026-05-22T13:39:56Z"
status: "True"
type: Ready
- lastTransitionTime: "2026-05-22T13:39:56Z"
status: "True"
type: Available
- lastTransitionTime: "2026-05-22T15:08:00Z"
reason: WaitingForMachineSet
severity: Info
status: "False"
type: MachineSetReady
readyReplicas: 1
unavailableReplicas: 1
updatedReplicas: 0
observedGeneration: 3
phase: Running
readyReplicas: 1
replicas: 1
selector: cluster.x-k8s.io/cluster-name=workload-docker-01,cluster.x-k8s.io/deployment-name=workload-docker-01-wave-1
upToDateReplicas: 0
```

### What did you expect to happen?

I would expect MD status to be always consistent regardless of which change is triggering a re-roll.

### Cluster API version

1.13.2

### Kubernetes version

1.34.0

### Anything else you would like to add?

_No response_

### Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.