kubeslice / kubeslice/worker-operator

Worker observability: expose controller-connection health and last-seen

Open
#469 0 comments 0 reactions 1 assignee Claimed by @gourishkb View on GitHub
enhancement feature lfx-mentorship
Dominant language
Go
Stars
62
Forks
33
Avg merge
3d 10h
Merged PRs (30d)
3

Description

## Summary

Add conditions, events, and metrics to the worker operator so that platform operators can observe controller-connection health and track failover progress in real time.

**Depends on:** #467 (worker reconnect implemented)

---

## Conditions

Add the following conditions to the `Cluster` CR (or equivalent worker status CR — confirm against ADR-001 Decision 7):

```yaml
status:
conditions:
- type: ControllerConnected
status: "True" # or "False" or "Unknown"
reason: Connected # see reason table below
message: ""
lastTransitionTime: "2026-06-24T10:00:00Z"
- type: ControllerEndpointSynced
status: "True"
reason: EndpointUpToDate
message: ""
lastTransitionTime: "2026-06-24T10:00:00Z"
```

**Reason codes for `ControllerConnected`:**

| Reason | Status | Meaning |
|---|---|---|
| `Connected` | True | Management-plane connection to hub is healthy |
| `ReconnectedAfterFailover` | True | Successfully reconnected after hub endpoint changed |
| `DialFailed` | False | Could not reach controller endpoint; retrying |
| `CertVerificationFailed` | False | TLS handshake failed — CA bundle mismatch |
| `EndpointNotConfigured` | Unknown | No controllerEndpoint field present (non-HA mode) |
| `Reconnecting` | Unknown | Endpoint changed; dial in progress |

Condition transitions must be idempotent — calling `SetCondition` with the same values must not create a spurious update.

---

## Events

Emit Events on the worker namespace for these transitions:

| `reason` | `type` | When |
|---|---|---|
| `ControllerEndpointChanged` | Normal | Cluster CR `controllerEndpoint` field updated |
| `ControllerConnected` | Normal | Connection to hub established or re-established |
| `ControllerConnectionLost` | Warning | Connection dropped; reconnect in progress |
| `CertVerificationFailed` | Warning | TLS handshake failed with new endpoint |

---

## Metrics (optional, implement if time permits)

```
# HELP worker_controller_reconnect_attempts_total Number of connection attempts to hub controller
# TYPE worker_controller_reconnect_attempts_total counter
worker_controller_reconnect_attempts_total{result="success"} 1
worker_controller_reconnect_attempts_total{result="failure"} 3

# HELP worker_controller_last_sync_time_seconds Unix timestamp of last successful sync from hub
# TYPE worker_controller_last_sync_time_seconds gauge
worker_controller_last_sync_time_seconds 1750000000
```

---

## Acceptance Criteria

- [ ] `ControllerConnected` condition present on `Cluster` CR after worker starts
- [ ] Condition transitions to `False/DialFailed` when hub is unreachable — verified by unit test with fake dialer
- [ ] Condition transitions to `True/ReconnectedAfterFailover` after reconnect — verified by unit test
- [ ] Condition transitions are idempotent (no spurious updates on no-op)
- [ ] All four Events above are emitted in the correct scenarios
- [ ] `reason` values are consistent with the table above — no ad hoc strings elsewhere in the code

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.