devantler-tech / devantler-tech/ksail
feat(talos): add an explicit guarded single-node replacement operation
- Dominant language
- Go
- Stars
- 165
- Forks
- 12
- Avg merge
- 5h 41m
- Merged PRs (30d)
- 337
Description
> 🤖 Generated by the Agentic Engineer
Talos on Hetzner currently replaces nodes only when `cluster update` detects a server-type change. That operation selects every mismatching node in a role; it cannot express a deliberate single-node replacement with unchanged desired configuration. Operators cannot use it for a bounded recovery drill without inventing configuration drift.
The existing rolling implementation also has prerequisites that must be strengthened before a new command can safely expose it: it checks inventory count rather than healthy etcd survivors, prepares the boot image after deleting the outgoing server, permits a missing Kubernetes node, and accepts replacement readiness by a reusable name or IP. The ordinary `cluster backup` archive contains manifests, not an etcd snapshot. #6931 / #6932 address the separate requirement that failed membership cleanup never authorizes deletion.
Acceptance criteria:
- Add an experimental, default-off operation that takes exactly one explicit node and binds the plan to cluster ownership, immutable Hetzner server ID, Kubernetes Node UID, and etcd member/cluster identity. Reject absent, ambiguous, changed, or mismatched identities before mutation.
- Provide a read-only plan/preflight mode. Apply must re-read identities and safety facts immediately before each irreversible boundary; a stale plan is rejected.
- Prove a healthy surviving etcd quorum using authenticated member/status observations, with consistent membership and no alarms. Provider inventory count and Kubernetes Ready alone are insufficient.
- Prepare and validate the exact boot image and role configuration before cordon or membership mutation. Capture a fresh verified etcd snapshot and the bootstrap/config material needed to rejoin from surviving control planes into a private recovery artifact; do not log secret material.
- Respect PodDisruptionBudgets. Require successful target-member removal before server destruction, preserving the #6932 failure behavior. Never fall through to another target or blindly retry an ambiguous membership mutation.
- Recover the stable API endpoint using a surviving control plane when needed. Confirm the original kubeconfig still works and accept completion only for a new provider ID and Node UID with the expected membership and Ready state.
- Record bounded recoverable phase evidence. A failure after membership removal must identify the retained/replacement state without automatically resetting or deleting unrelated resources.
- A consuming production workflow must be manual/default-off, share its existing deployment-concurrency lock, and pin a reviewed released KSail version containing every required guard. The source change alone does not authorize a production drill.
- Cover rejection paths with real callers and no-mutation assertions, successful ordering with injected provider/Talos/Kubernetes clients, cancellation/ambiguous errors, both experimental-gate states, and a disposable-cluster integration proof before a production acceptance drill.
Implementation should proceed in reviewable prerequisites, beginning with preparing replacement boot/config material before destructive actions. This issue tracks agent-owned engineering work; it is not blocked on a maintainer running raw deletion commands.
Context: https://github.com/devantler-tech/platform/issues/2120 and https://github.com/devantler-tech/platform/issues/2764.
Contributor guide
Research direction
Start by tracing the existing Talos `cluster update` rolling implementation and the `cluster backup` behavior, then map the provider, Talos, Kubernetes, and etcd client entry points involved. Review the failure behavior in #6931 and #6932. Done means a gated read-only and apply workflow enforces identity, quorum, ordering, recovery evidence, locking, and the listed unit, injected-client, cancellation, and disposable-cluster tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- cli, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100