devantler-tech / devantler-tech/ksail

feat(talos): add an explicit guarded single-node replacement operation

Open
#6,933 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
165
Forks
12
Avg merge
5h 41m
Merged PRs (30d)
337

Description

> 🤖 Generated by the Agentic Engineer

Talos on Hetzner currently replaces nodes only when `cluster update` detects a server-type change. That operation selects every mismatching node in a role; it cannot express a deliberate single-node replacement with unchanged desired configuration. Operators cannot use it for a bounded recovery drill without inventing configuration drift.

The existing rolling implementation also has prerequisites that must be strengthened before a new command can safely expose it: it checks inventory count rather than healthy etcd survivors, prepares the boot image after deleting the outgoing server, permits a missing Kubernetes node, and accepts replacement readiness by a reusable name or IP. The ordinary `cluster backup` archive contains manifests, not an etcd snapshot. #6931 / #6932 address the separate requirement that failed membership cleanup never authorizes deletion.

Acceptance criteria:

- Add an experimental, default-off operation that takes exactly one explicit node and binds the plan to cluster ownership, immutable Hetzner server ID, Kubernetes Node UID, and etcd member/cluster identity. Reject absent, ambiguous, changed, or mismatched identities before mutation.
- Provide a read-only plan/preflight mode. Apply must re-read identities and safety facts immediately before each irreversible boundary; a stale plan is rejected.
- Prove a healthy surviving etcd quorum using authenticated member/status observations, with consistent membership and no alarms. Provider inventory count and Kubernetes Ready alone are insufficient.
- Prepare and validate the exact boot image and role configuration before cordon or membership mutation. Capture a fresh verified etcd snapshot and the bootstrap/config material needed to rejoin from surviving control planes into a private recovery artifact; do not log secret material.
- Respect PodDisruptionBudgets. Require successful target-member removal before server destruction, preserving the #6932 failure behavior. Never fall through to another target or blindly retry an ambiguous membership mutation.
- Recover the stable API endpoint using a surviving control plane when needed. Confirm the original kubeconfig still works and accept completion only for a new provider ID and Node UID with the expected membership and Ready state.
- Record bounded recoverable phase evidence. A failure after membership removal must identify the retained/replacement state without automatically resetting or deleting unrelated resources.
- A consuming production workflow must be manual/default-off, share its existing deployment-concurrency lock, and pin a reviewed released KSail version containing every required guard. The source change alone does not authorize a production drill.
- Cover rejection paths with real callers and no-mutation assertions, successful ordering with injected provider/Talos/Kubernetes clients, cancellation/ambiguous errors, both experimental-gate states, and a disposable-cluster integration proof before a production acceptance drill.

Implementation should proceed in reviewable prerequisites, beginning with preparing replacement boot/config material before destructive actions. This issue tracks agent-owned engineering work; it is not blocked on a maintainer running raw deletion commands.

Context: https://github.com/devantler-tech/platform/issues/2120 and https://github.com/devantler-tech/platform/issues/2764.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing Talos `cluster update` rolling implementation and the `cluster backup` behavior, then map the provider, Talos, Kubernetes, and etcd client entry points involved. Review the failure behavior in #6931 and #6932. Done means a gated read-only and apply workflow enforces identity, quorum, ordering, recovery evidence, locking, and the listed unit, injected-client, cancellation, and disposable-cluster tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
cli, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.