kubeslice / kubeslice/kubeslice-controller

Observability + runbook for HA (metrics/events/docs)

Open
#298 0 comments 0 reactions 1 assignee Claimed by @gourishkb View on GitHub
feature lfx-mentorship
Dominant language
Go
Stars
73
Forks
48
Avg merge
2d 21h
Merged PRs (30d)
8

Description

## Summary

Add Prometheus metrics, Kubernetes Events, and an operator runbook for the HA Active/Standby system.

**Depends on:** #294, #295, #297 (all implementation issues complete)

---

## Metrics

Register all metrics in `pkg/ha/metrics.go` using the standard `controller-runtime` metrics registry.

| Metric name | Type | Labels | Description |
|---|---|---|---|
| `ha_leader_status` | Gauge | — | 1 = this instance is Active, 0 = Standby |
| `ha_lease_last_renew_time_seconds` | Gauge | — | Unix timestamp of last successful Lease renewal (Active only) |
| `ha_sync_lag_seconds` | Histogram | `kind` | Lag between object change on Active and mirror write on Standby |
| `ha_sync_errors_total` | Counter | `kind`, `operation` | Sync failures per CRD type and operation |
| `ha_failover_total` | Counter | — | Number of completed promotions since process start |

Buckets for `ha_sync_lag_seconds`: `[0.1, 0.5, 1, 2, 5, 10, 30]` seconds.

---

## Kubernetes Events

Emit Events on the controller Pod or the `kubeslice-system` namespace for these transitions:

| `reason` | `type` | When |
|---|---|---|
| `BecameActive` | Normal | Controller starts in Active mode |
| `BecameStandby` | Normal | Controller starts in Standby mode |
| `PromotedToActive` | Normal | Standby successfully completes promotion sequence |
| `LeadershipLost` | Warning | Active failed to renew Lease N consecutive times |
| `SyncError` | Warning | RemoteSyncer failed to apply an object to Standby |
| `PromotionAborted` | Warning | Promotion attempt was aborted (split-brain guard fired) |

---

## Runbook (docs/ha-runbook.md)

The runbook must cover these scenarios:

### 1. Verify Active/Standby status
```bash
# Check leader status metric on each hub cluster
kubectl -n kubeslice-system port-forward svc/kubeslice-controller-metrics 8080
curl -s localhost:8080/metrics | grep ha_leader_status

# Check Events for recent transitions
kubectl -n kubeslice-system get events --field-selector reason=PromotedToActive
```

### 2. Simulate failover (test procedure)
```bash
# Scale down Active controller
kubectl --context hub-active -n kubeslice-system scale deploy/kubeslice-controller --replicas=0

# Watch Standby logs for promotion
kubectl --context hub-standby -n kubeslice-system logs -f deploy/kubeslice-controller | grep -i promot

# Verify workers reconnected
kubectl --context worker-1 -n kubeslice-system get cluster -o jsonpath='{.items[*].status.conditions}'
```

### 3. Credential rotation (Active kubeconfig Secret)
Steps to update `ha-active-kubeconfig` Secret on Standby and restart the controller without losing sync.

### 4. Troubleshooting: sync lag high
- Check `ha_sync_lag_seconds` histogram
- Check `ha_sync_errors_total` for errors
- Check network connectivity between hub clusters

### 5. Troubleshooting: promotion not firing
- Verify `--ha-mode=standby` flag is set
- Verify `ha-active-kubeconfig` Secret exists and is valid
- Check Standby logs for `WatchRemoteLease` errors

---

## Acceptance Criteria

- [ ] All five metrics above are registered and emitted; verify with `/metrics` endpoint
- [ ] All six Event reasons are emitted in the correct scenarios (verified by unit test using fake recorder)
- [ ] Runbook covers all five scenarios above with working `kubectl` commands
- [ ] Metrics have `HELP` and `TYPE` comments in the `/metrics` output

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.