kubeslice / kubeslice/kubeslice-controller
Observability + runbook for HA (metrics/events/docs)
- Dominant language
- Go
- Stars
- 73
- Forks
- 48
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 8
Description
## Summary
Add Prometheus metrics, Kubernetes Events, and an operator runbook for the HA Active/Standby system.
**Depends on:** #294, #295, #297 (all implementation issues complete)
---
## Metrics
Register all metrics in `pkg/ha/metrics.go` using the standard `controller-runtime` metrics registry.
| Metric name | Type | Labels | Description |
|---|---|---|---|
| `ha_leader_status` | Gauge | — | 1 = this instance is Active, 0 = Standby |
| `ha_lease_last_renew_time_seconds` | Gauge | — | Unix timestamp of last successful Lease renewal (Active only) |
| `ha_sync_lag_seconds` | Histogram | `kind` | Lag between object change on Active and mirror write on Standby |
| `ha_sync_errors_total` | Counter | `kind`, `operation` | Sync failures per CRD type and operation |
| `ha_failover_total` | Counter | — | Number of completed promotions since process start |
Buckets for `ha_sync_lag_seconds`: `[0.1, 0.5, 1, 2, 5, 10, 30]` seconds.
---
## Kubernetes Events
Emit Events on the controller Pod or the `kubeslice-system` namespace for these transitions:
| `reason` | `type` | When |
|---|---|---|
| `BecameActive` | Normal | Controller starts in Active mode |
| `BecameStandby` | Normal | Controller starts in Standby mode |
| `PromotedToActive` | Normal | Standby successfully completes promotion sequence |
| `LeadershipLost` | Warning | Active failed to renew Lease N consecutive times |
| `SyncError` | Warning | RemoteSyncer failed to apply an object to Standby |
| `PromotionAborted` | Warning | Promotion attempt was aborted (split-brain guard fired) |
---
## Runbook (docs/ha-runbook.md)
The runbook must cover these scenarios:
### 1. Verify Active/Standby status
```bash
# Check leader status metric on each hub cluster
kubectl -n kubeslice-system port-forward svc/kubeslice-controller-metrics 8080
curl -s localhost:8080/metrics | grep ha_leader_status
# Check Events for recent transitions
kubectl -n kubeslice-system get events --field-selector reason=PromotedToActive
```
### 2. Simulate failover (test procedure)
```bash
# Scale down Active controller
kubectl --context hub-active -n kubeslice-system scale deploy/kubeslice-controller --replicas=0
# Watch Standby logs for promotion
kubectl --context hub-standby -n kubeslice-system logs -f deploy/kubeslice-controller | grep -i promot
# Verify workers reconnected
kubectl --context worker-1 -n kubeslice-system get cluster -o jsonpath='{.items[*].status.conditions}'
```
### 3. Credential rotation (Active kubeconfig Secret)
Steps to update `ha-active-kubeconfig` Secret on Standby and restart the controller without losing sync.
### 4. Troubleshooting: sync lag high
- Check `ha_sync_lag_seconds` histogram
- Check `ha_sync_errors_total` for errors
- Check network connectivity between hub clusters
### 5. Troubleshooting: promotion not firing
- Verify `--ha-mode=standby` flag is set
- Verify `ha-active-kubeconfig` Secret exists and is valid
- Check Standby logs for `WatchRemoteLease` errors
---
## Acceptance Criteria
- [ ] All five metrics above are registered and emitted; verify with `/metrics` endpoint
- [ ] All six Event reasons are emitted in the correct scenarios (verified by unit test using fake recorder)
- [ ] Runbook covers all five scenarios above with working `kubectl` commands
- [ ] Metrics have `HELP` and `TYPE` comments in the `/metrics` output
Contributor guide
Assessment
This issue has not been assessed yet.