kubernetes-sigs / kubernetes-sigs/cluster-api

Add a fail-closed in-place update policy and make Machine replacement decisions predictable

Open
#13,863 2 comments 0 reactions 0 assignees View on GitHub
kind/feature needs-priority needs-triage
Dominant language
Go
Stars
4.3k
Forks
1.6k
Avg merge
1d 3h
Merged PRs (30d)
113

Description

## What would you like to be added (User Story)?

As a Cluster API user/operator or a platform built on top of Cluster API, I would like to require that a rollout is performed only via in-place updates, or otherwise stops without replacing existing Machines.

I would also like to be able to reliably know whether a given change is expected to trigger in-place updates or Machine replacement before the rollout actually replaces Machines.

This is important for environments where replacing the underlying Machine is disruptive or unacceptable, for example nodes with local storage, bare metal clusters, or systems where machine identity must be preserved.

For users who explicitly adopt in-place updates, preserving existing Machines can be a stronger requirement than making the cluster converge to the new desired state. In those cases, if the desired state cannot be achieved without replacing Machines, the safer and expected behavior is to stop and report why, rather than automatically falling back to Machine replacement.

## Detailed Description

Cluster API now supports in-place updates via Runtime SDK hooks. However, the current model makes in-place update an internal decision owned by Cluster API.

Runtime extensions can report which changes they are able to handle in-place, but the final decision still depends on Cluster API internals, including diff calculation, extension responses, eligibility checks, preflight checks, health state, rollout budgets, controller state, and fallback behavior.

This creates two related gaps:

1. Users cannot express a hard policy like:

> Apply this change only if it can be completed in-place. If it cannot be completed in-place, do not delete or replace any existing Machine; stop the rollout and surface an explicit condition/reason.

2. Users and higher-level platforms cannot reliably predict whether a given change will result in in-place updates or rolling replacement.

A platform can try to approximate CAPI's decision by reproducing the diff and extension logic, but this is not a stable contract and can still diverge from the actual controller decision because runtime state, health, preflight checks, rollout budgets, and reconciliation timing can change the outcome.

The current proposal and documentation say that users should care about desired state and Cluster API chooses the best strategy to achieve it. They also say that if external update extensions cannot cover all required changes, CAPI falls back to immutable/rolling rollouts.

This is a reasonable default behavior, but it does not cover users for whom Machine preservation is a hard safety constraint. For those users, automatically falling back to Machine replacement can be worse than leaving the cluster temporarily not converged to the new desired state.

For some environments, "CAPI may replace the Machine if in-place update is not possible" is not acceptable. The required behavior is:

> Unless the user explicitly allows replacement, CAPI must not replace existing Machines.

## Possible approaches

This issue does not intend to prescribe a specific API, but possible directions could include:

1. Add a fail-closed rollout policy for KCP / MachineDeployment, for example `InPlaceOnly` or `NeverReplace`.
- If all required changes can be handled in-place, proceed.
- If any Machine would require replacement, stop the rollout and set a condition explaining why.

2. Expose a planned rollout decision or dry-run style status/API.
- Users and higher-level platforms could see whether the current desired state is expected to use in-place update or replacement.
- This would improve predictability even if Cluster API keeps the final decision internally.

3. Add explicit conditions/events when CAPI decides that in-place update is not possible or when it falls back to rolling replacement.
- Example reasons: unsupported diff, no extension, extension declined, failed preflight check, unhealthy Machine, rollout budget constraints.

4. Provide a policy hook that allows platforms to reject Machine replacement.
- This would let platform owners enforce "no replacement" policies without each provider inventing its own mechanism.

## Why this matters

Without such a policy or prediction mechanism, enabling in-place updates does not guarantee that Machines will be preserved, and users cannot reliably know how a change will be executed.

This makes in-place updates hard to use for platforms that need to provide strong operational guarantees to end users.

More importantly, for some in-place update users, preserving existing Machines is not just an optimization. It is the primary safety requirement. In those environments, failing closed is preferable to converging by replacing Machines.

## Related context

- In-place updates proposal: https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20240807-in-place-updates.md
- Runtime SDK in-place update hooks documentation: https://cluster-api.sigs.k8s.io/tasks/experimental-features/runtime-sdk/implement-in-place-update-hooks
- In-place updates tracking issue: https://github.com/kubernetes-sigs/cluster-api/issues/12975

## Label(s) to be applied

/kind feature
/area/runtime-sdk
/area/machinedeployment
/area/provider/control-plane-kubeadm

Contributor guide

Open the contributing guide

Research direction

Start by reading the in-place updates proposal and the Runtime SDK in-place update hooks documentation linked in the issue. Then review the related tracking issue and the KCP/MachineDeployment behavior described here. Done means agreeing on a concrete API or policy design that can fail closed, expose the rollout decision, and report why replacement would occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend-api-design, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.