kubeslice / kubeslice/worker-operator
Worker observability: expose controller-connection health and last-seen
- Dominant language
- Go
- Stars
- 62
- Forks
- 33
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 3
Description
## Summary
Add conditions, events, and metrics to the worker operator so that platform operators can observe controller-connection health and track failover progress in real time.
**Depends on:** #467 (worker reconnect implemented)
---
## Conditions
Add the following conditions to the `Cluster` CR (or equivalent worker status CR — confirm against ADR-001 Decision 7):
```yaml
status:
conditions:
- type: ControllerConnected
status: "True" # or "False" or "Unknown"
reason: Connected # see reason table below
message: ""
lastTransitionTime: "2026-06-24T10:00:00Z"
- type: ControllerEndpointSynced
status: "True"
reason: EndpointUpToDate
message: ""
lastTransitionTime: "2026-06-24T10:00:00Z"
```
**Reason codes for `ControllerConnected`:**
| Reason | Status | Meaning |
|---|---|---|
| `Connected` | True | Management-plane connection to hub is healthy |
| `ReconnectedAfterFailover` | True | Successfully reconnected after hub endpoint changed |
| `DialFailed` | False | Could not reach controller endpoint; retrying |
| `CertVerificationFailed` | False | TLS handshake failed — CA bundle mismatch |
| `EndpointNotConfigured` | Unknown | No controllerEndpoint field present (non-HA mode) |
| `Reconnecting` | Unknown | Endpoint changed; dial in progress |
Condition transitions must be idempotent — calling `SetCondition` with the same values must not create a spurious update.
---
## Events
Emit Events on the worker namespace for these transitions:
| `reason` | `type` | When |
|---|---|---|
| `ControllerEndpointChanged` | Normal | Cluster CR `controllerEndpoint` field updated |
| `ControllerConnected` | Normal | Connection to hub established or re-established |
| `ControllerConnectionLost` | Warning | Connection dropped; reconnect in progress |
| `CertVerificationFailed` | Warning | TLS handshake failed with new endpoint |
---
## Metrics (optional, implement if time permits)
```
# HELP worker_controller_reconnect_attempts_total Number of connection attempts to hub controller
# TYPE worker_controller_reconnect_attempts_total counter
worker_controller_reconnect_attempts_total{result="success"} 1
worker_controller_reconnect_attempts_total{result="failure"} 3
# HELP worker_controller_last_sync_time_seconds Unix timestamp of last successful sync from hub
# TYPE worker_controller_last_sync_time_seconds gauge
worker_controller_last_sync_time_seconds 1750000000
```
---
## Acceptance Criteria
- [ ] `ControllerConnected` condition present on `Cluster` CR after worker starts
- [ ] Condition transitions to `False/DialFailed` when hub is unreachable — verified by unit test with fake dialer
- [ ] Condition transitions to `True/ReconnectedAfterFailover` after reconnect — verified by unit test
- [ ] Condition transitions are idempotent (no spurious updates on no-op)
- [ ] All four Events above are emitted in the correct scenarios
- [ ] `reason` values are consistent with the table above — no ad hoc strings elsewhere in the code
Contributor guide
Assessment
This issue has not been assessed yet.