LFX Mentorship: HA Active/Standby project

## Description

Add **Active/Standby HA support for the KubeSlice Controller across two separate hub clusters** (multi-cluster HA — not multi-node/pod-replica HA).

- **Leader election:** Only one controller *cluster* (the Active hub) holds a Lease and is permitted to write updates to worker clusters or manage Slice configurations.
- **State sync:** The Standby cluster continuously mirrors the Active cluster's KubeSlice CRDs (Slices, SliceConfigs, SliceGateways, ServiceExports, Clusters, WorkerSliceConfigs) via a remote informer.
- **Heartbeating:** The Standby watches the Active cluster's Lease renewal via a remote client. Stale renewal beyond a configurable threshold triggers promotion.
- **Failover:** Active fails to renew its Lease → Standby detects timeout → acquires leadership on its own cluster → promotes itself → workers are re-pointed to the new Active controller.

**Expected Outcome:** An Active/Standby HA architecture that makes the KubeSlice management plane resilient and disaster-recovery-ready.

**Recommended Skills:** Go, Kubernetes (controllers, CRDs, client-go)

---

## Architecture Clarification — Multi-Cluster, Not Multi-Node

> **Important for contributors:** The existing `--leader-elect` flag and `LeaderElectionID` in `main.go` handle **in-cluster pod-level** leader election (multiple pods of the same controller competing within one cluster). That mechanism is **not** what this project implements. This project requires **cross-cluster** leader election between two separate Kubernetes clusters. The existing flag is a red herring — do not base your design on enabling it.

### Topology

```
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ Active Hub Cluster (K8s) │ │ Standby Hub Cluster (K8s) │
│ │ │ │
│ kubeslice-controller │ │ kubeslice-controller │
│ ├─ Reconcilers: ENABLED │ │ ├─ Reconcilers: GATED (no-ops) │
│ ├─ Holds Lease CR (local) │◄────│ ├─ RemoteWatcher → Active API │
│ └─ Writes to worker clusters │ │ ├─ RemoteSyncer → mirrors CRDs │
│ │ │ └─ Promotes when Lease expires │
└─────────────────┬────────────────┘ └──────────────────────────────────┘
│ endpoint / certs (re-pointed on failover)
┌───────────┼───────────┐
▼ ▼ ▼
[Worker 1] [Worker 2] [Worker N]
worker-operator (reconciles new controller endpoint on failover)
```

### What the Standby needs

1. **Remote client to Active hub cluster** — kubeconfig stored as a K8s Secret in the Standby cluster, mounted into the controller pod. Used to watch the Lease and run informers over mirrored CRDs.
2. **Remote clients to each worker cluster** — so on promotion the new Active can immediately resume writing SliceConfigs.

### Fencing model

Active gates **every mutating reconciler call** on local Lease validity at runtime (not once at startup). Standby reconcilers return immediately without writing. If Active's own API server goes down, it cannot renew its Lease and cannot write — natural fencing.

**Split-brain (explicit non-goal for MVP):** If a network partition isolates the two hub clusters from each other but both remain healthy, Active keeps renewing its local Lease while Standby can no longer see it and promotes. Both would write to workers. This scenario is out of scope for the initial implementation and must be documented as a known limitation in the ADR.

---

## Implementation Guide (per downstream issue)

### #293 — ADR (prerequisite, must merge before any code)

The ADR must answer all of the following before it is approved:

| Question | Required answer |
|---|---|
| Where does the Lease live? | On the Active cluster's own API server (not external). Standby watches via remote client. |
| How does Standby get credentials to Active? | kubeconfig as a K8s Secret in Standby, mounted into controller pod. |
| What triggers promotion? | Remote Lease `renewTime` not updated within `leaseDuration + paddingSeconds`. |
| Fencing model? | Active gates each reconcile on local Lease validity. Dead Active = can't renew = can't write. |
| Split-brain? | Explicit non-goal. Document: network partition between hub clusters may cause dual-write. |
| Which CRDs are mirrored Active → Standby? | Enumerate: Slices, SliceConfigs, SliceGateways, ServiceExports, Clusters, WorkerSliceConfigs. |
| Worker discovery of new Active? | CR-driven: `ClusterController` CR on each worker holds `endpoint + caBundle`; new Active updates it on promotion. |

Deliverables: components diagram, normal-operation sequence diagram, failover sequence diagram, non-goals list.

### #294 — Leader election + write fencing

**Do not use `--leader-elect`** (in-cluster pod gate). Build a `ClusterLeaderElector`:

```go
// pkg/ha/leader_elector.go
type ClusterLeaderElector struct {
localClient client.Client // own cluster — renew Lease
remoteClient client.Client // Standby only — watch Active's Lease
mode HAMode // Active | Standby | Standalone
}

func (e *ClusterLeaderElector) IsLeader() bool
func (e *ClusterLeaderElector) StartLeaseRenewal(ctx context.Context) // Active only
func (e *ClusterLeaderElector) WatchRemoteLease(ctx context.Context) // Standby only
```

Every reconciler's `Reconcile()` method must begin with:

```go
if !r.leaderElector.IsLeader() {
log.Info("standby mode, skipping write")
return ctrl.Result{}, nil
}
```

Acceptance: Standby logs "standby mode, skipping write" on every reconciler invocation — verified by test. `IsLeader()` called on every loop, not once at startup.

### #295 — State sync (Active → Standby)

New goroutine running on Standby only:

```go
// pkg/ha/remote_syncer.go
type RemoteSyncer struct {
sourceClient client.Client // to Active cluster
targetClient client.Client // to own (Standby) cluster
}
```

Runs informers on `sourceClient` for each mirrored type. Must handle all three event paths:
- **Create:** CR exists on Active but not Standby → create on Standby (strip `resourceVersion`, `uid`)
- **Update:** CR changed on Active → update on Standby
- **Delete:** CR deleted on Active → delete on Standby

Expose `ha_sync_lag_seconds` metric = time between CR creation/update on Active and its appearance on Standby.

### #297 — Failover / promotion logic

Promotion sequence:
1. `WatchRemoteLease` detects stale `renewTime` beyond threshold
2. One final dial to Active API server — if reachable and Lease is live, abort (transient blip)
3. Acquire Lease on own cluster
4. Set `mode = Active`, enable reconcilers, stop `RemoteSyncer`
5. Update `ClusterController` CR on all worker clusters with new endpoint
6. Emit K8s Event + increment `ha_failover_total` metric

Configurable: `leaseDuration`, `renewDeadline`, `retryPeriod`, `promotionGracePeriod`.

### worker-operator #467 — Worker reconnection

Worker operator watches the `ClusterController` CR for changes to `endpoint` and `caBundle`. On change: gracefully close existing connection, dial new endpoint, validate cert trust, reconcile status conditions. Must be backward-compatible — non-HA deployments see no behavior change (CR field absent or static).

---

## Environment Setup (required before Week 3 code work)

Contributors must have a working 3-cluster Kind setup before implementing #294:

```bash
# Three clusters
kind create cluster --name hub-active
kind create cluster --name hub-standby
kind create cluster --name worker-1

# Standby needs a kubeconfig Secret to reach hub-active
kubectl --context kind-hub-active config view --minify --flatten > /tmp/hub-active-kubeconfig.yaml
kubectl --context kind-hub-standby create secret generic hub-active-kubeconfig \
--from-file=kubeconfig=/tmp/hub-active-kubeconfig.yaml
```

Mount this Secret into the Standby controller pod and construct `remoteClient` from it at startup.

---

## Delivery Timeline (June 24 – August 31)

| Week | Dates | Deliverable |
|---|---|---|
| 1 | Jun 24 – Jun 30 | ADR (#293) drafted and in review |
| 2 | Jul 1 – Jul 7 | ADR merged; 3-cluster Kind env working; remote client dials Active |
| 3 | Jul 8 – Jul 14 | #294: `ClusterLeaderElector` + reconciler fencing (demo in Kind) |
| 4 | Jul 15 – Jul 21 | #295: `RemoteSyncer` — all 3 event types tested |
| **Midterm** | **~Jul 21** | **Demo: create Slice on Active → appears on Standby; kill Active → Standby promotes** |
| 5 | Jul 22 – Jul 28 | #297: Promotion logic end-to-end in Kind |
| 6 | Jul 29 – Aug 4 | worker-op #467: Worker reconciles new controller endpoint |
| 7 | Aug 5 – Aug 11 | #298: Metrics + runbook; worker-op #469: connection health conditions |
| 8 | Aug 12 – Aug 18 | #299 + worker-op #468: E2E + failover robustness tests in CI |
| 9–10 | Aug 19 – Aug 31 | PR merge queue, final reviews, LFX evaluation submission |

**Midterm demo criteria (July 21):**
1. `hub-standby` controller logs "standby mode" on every reconcile attempt
2. Create a Slice on `hub-active` → verify it appears on `hub-standby` (CRD mirror working)
3. Kill `hub-active` controller pod → `hub-standby` logs "promoting to Active" within `leaseDuration`
4. New Slice created post-promotion is reconciled successfully by `hub-standby`

---

## Mentors

- Gourish Biradar (email: biradar.gourish@gmail.com, github: gourishkb)
- Prabhu Navali (email: prabhu@avesha.io, github: pnavali)
- Rahul Kumar (email: rahulparida933@gmail.com, github: Rahul-D78)

## Downstream Issues

**kubeslice-controller:**
- https://github.com/kubeslice/kubeslice-controller/issues/293
- https://github.com/kubeslice/kubeslice-controller/issues/294
- https://github.com/kubeslice/kubeslice-controller/issues/295
- https://github.com/kubeslice/kubeslice-controller/issues/297
- https://github.com/kubeslice/kubeslice-controller/issues/298
- https://github.com/kubeslice/kubeslice-controller/issues/299

**worker-operator:**
- https://github.com/kubeslice/worker-operator/issues/467
- https://github.com/kubeslice/worker-operator/issues/468
- https://github.com/kubeslice/worker-operator/issues/469

Start with downstream issue #293 and its ADR requirements, then review main.go and the proposed pkg/ha/leader_elector.go and pkg/ha/remote_syncer.go entry points. Use the three-cluster Kind setup to validate the staged work: remote CRD sync, reconciler fencing, lease-based promotion, worker re-pointing, and documented split-brain limitations.

Written by the indexing model from the issue text.