kubeslice / kubeslice/worker-operator
Feature: support controller endpoint/identity updates on failover
- Dominant language
- Go
- Stars
- 62
- Forks
- 33
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 3
Description
## Summary
Ensure worker clusters can automatically re-point to the new Active hub controller after failover — no manual steps, no traffic-plane disruption.
**Depends on:** kubeslice-controller #293 (ADR Decision 7 defines the CR carrier), #297 (promotion writes new endpoint to worker CR)
---
## Mechanism
On promotion, the new Active controller updates a CR on each worker cluster with the new hub endpoint and CA bundle. The worker operator watches this CR and reconciles its connection.
The exact CR field names must match the decision made in ADR-001 Decision 7. Placeholder names used here (confirm against ADR before coding):
```yaml
# Proposed fields on existing Cluster CR (or equivalent)
spec:
controllerEndpoint: "https://hub-standby.example.com:6443"
controllerCABundle:
```
---
## Worker reconcile behavior
When `controllerEndpoint` or `controllerCABundle` changes on the watched CR:
```
1. Detect change in Cluster CR spec (controller-runtime watch)
2. Close existing gRPC/HTTP connection to old hub endpoint
3. Build new TLS config from updated controllerCABundle
4. Dial new controllerEndpoint; retry with backoff up to --ha-reconnect-timeout (default: 60s)
5. On success: update condition ControllerConnected=True, reason=ReconnectedAfterFailover
6. On failure: update condition ControllerConnected=False, reason=DialFailed, message=
7. Emit K8s Event reason=ControllerEndpointChanged on the worker namespace
```
Connection close and redial must not affect the data plane (VPN tunnels, service routes). Only the management-plane connection to the hub is changed.
---
## Backward compatibility
- If `controllerEndpoint` field is absent or empty → no behavior change (non-HA deployment)
- Worker operator must not crash or error-loop on a CR that lacks these fields
- Existing workers running an older version without this feature must degrade gracefully (they will not auto-reconnect; operator must restart manually — document this in runbook)
---
## Acceptance Criteria
- [ ] Worker reconciler watches for changes to `controllerEndpoint` on the Cluster CR (or whichever CR is decided in ADR-001)
- [ ] On endpoint change: existing connection closed, new connection established with updated CA
- [ ] `ControllerConnected` condition updated within 5s of successful reconnect
- [ ] Unit test: fake client simulates endpoint update → verify dial to new endpoint is attempted
- [ ] Non-HA deployment (no field): zero behavior change — verified by existing test suite passing unchanged
- [ ] Demo: after failover in Kind, worker-1 reconnects to hub-standby without kubectl intervention
Contributor guide
Assessment
This issue has not been assessed yet.