kubeslice / kubeslice/kubeslice-controller
Failover/promotion logic: detect Active failure and promote Standby
- Dominant language
- Go
- Stars
- 73
- Forks
- 48
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 8
Description
## Summary
Implement detection of Active cluster failure and the promotion sequence that transitions the Standby into the Active role.
**Depends on:** #294 (`ClusterLeaderElector` in place), #295 (`RemoteSyncer` in place)
---
## Promotion sequence (Standby → Active)
These steps must execute in order:
```
1. RemoteWatcher detects Active Lease renewTime stale beyond (leaseDuration + promotionGrace)
2. Final liveness check: dial Active API server
→ if reachable AND Lease is live: abort promotion (transient blip)
→ if unreachable OR Lease expired: proceed
3. Acquire Lease on own (Standby) cluster — create if absent
4. Set leaderElector.mode = Active
5. Stop RemoteSyncer goroutine
6. Enable all reconcilers (IsLeader() now returns true)
7. For each worker cluster: update ClusterController CR with new hub endpoint + CA
8. Emit K8s Event reason=PromotedToActive on kubeslice-system namespace
9. Increment ha_failover_total metric
```
Step 7 must complete before reconcilers are considered fully operational. Surface a `PromotionInProgress` condition on the controller Pod or a status ConfigMap until step 7 finishes.
---
## Configuration flags
```
--ha-lease-duration duration 15s — How long a Lease is valid
--ha-renew-deadline duration 10s — How long Active retries renewal
--ha-retry-period duration 2s — Renewal retry interval
--ha-promotion-grace duration 5s — Extra buffer Standby waits before promoting
```
**Maximum failover time** (worst case): `leaseDuration + promotionGrace` = 15s + 5s = **20s** before promotion begins, plus worker update propagation (~2s per worker).
---
## Split-brain guard (step 2)
Before acquiring the Lease (step 3), make one final attempt to reach the Active cluster's API server. This prevents spurious promotion during a transient network blip.
```go
_, err := remoteClient.List(ctx, &coordinationv1.LeaseList{}, client.Limit(1))
if err == nil {
// Active API reachable — check if Lease is actually expired
lease := &coordinationv1.Lease{}
if err := remoteClient.Get(ctx, leaseKey, lease); err == nil {
if time.Since(lease.Spec.RenewTime.Time) < leaseDuration+promotionGrace {
// not actually expired — abort
return
}
}
}
// proceed with promotion
```
---
## Worker endpoint update (step 7)
For each worker cluster (Standby must have remote clients to all workers):
```go
cluster := &kubesliceiov1.Cluster{}
if err := workerClient.Get(ctx, clusterKey, cluster); err != nil { ... }
cluster.Spec.ControllerEndpoint = newActiveEndpoint
cluster.Spec.ControllerCABundle = newActiveCABundle
workerClient.Update(ctx, cluster)
```
The field names (`ControllerEndpoint`, `ControllerCABundle`) must match the decision in ADR-001 Decision 7 and must align with the implementation in worker-operator #467.
---
## Acceptance Criteria
- [ ] Unit test: `WatchRemoteLease` with a fake remote client whose Lease stops renewing → promotion sequence fires
- [ ] Unit test: transient blip (Lease briefly stale but still valid) → promotion aborted
- [ ] `ha_failover_total` counter increments on each successful promotion
- [ ] K8s Event `reason: PromotedToActive` emitted on promotion
- [ ] All configurable timeouts have sensible defaults and are documented
- [ ] Demo in Kind: kill Active controller pod → Standby promotes within `leaseDuration + promotionGrace`; new Slice created on Standby reconciles successfully
Contributor guide
Assessment
This issue has not been assessed yet.