kubeslice / kubeslice/worker-operator

Feature: support controller endpoint/identity updates on failover

Open
#467 0 comments 0 reactions 1 assignee Claimed by @gourishkb View on GitHub
enhancement feature lfx-mentorship
Dominant language
Go
Stars
62
Forks
33
Avg merge
3d 10h
Merged PRs (30d)
3

Description

## Summary

Ensure worker clusters can automatically re-point to the new Active hub controller after failover — no manual steps, no traffic-plane disruption.

**Depends on:** kubeslice-controller #293 (ADR Decision 7 defines the CR carrier), #297 (promotion writes new endpoint to worker CR)

---

## Mechanism

On promotion, the new Active controller updates a CR on each worker cluster with the new hub endpoint and CA bundle. The worker operator watches this CR and reconciles its connection.

The exact CR field names must match the decision made in ADR-001 Decision 7. Placeholder names used here (confirm against ADR before coding):

```yaml
# Proposed fields on existing Cluster CR (or equivalent)
spec:
controllerEndpoint: "https://hub-standby.example.com:6443"
controllerCABundle:
```

---

## Worker reconcile behavior

When `controllerEndpoint` or `controllerCABundle` changes on the watched CR:

```
1. Detect change in Cluster CR spec (controller-runtime watch)
2. Close existing gRPC/HTTP connection to old hub endpoint
3. Build new TLS config from updated controllerCABundle
4. Dial new controllerEndpoint; retry with backoff up to --ha-reconnect-timeout (default: 60s)
5. On success: update condition ControllerConnected=True, reason=ReconnectedAfterFailover
6. On failure: update condition ControllerConnected=False, reason=DialFailed, message=
7. Emit K8s Event reason=ControllerEndpointChanged on the worker namespace
```

Connection close and redial must not affect the data plane (VPN tunnels, service routes). Only the management-plane connection to the hub is changed.

---

## Backward compatibility

- If `controllerEndpoint` field is absent or empty → no behavior change (non-HA deployment)
- Worker operator must not crash or error-loop on a CR that lacks these fields
- Existing workers running an older version without this feature must degrade gracefully (they will not auto-reconnect; operator must restart manually — document this in runbook)

---

## Acceptance Criteria

- [ ] Worker reconciler watches for changes to `controllerEndpoint` on the Cluster CR (or whichever CR is decided in ADR-001)
- [ ] On endpoint change: existing connection closed, new connection established with updated CA
- [ ] `ControllerConnected` condition updated within 5s of successful reconnect
- [ ] Unit test: fake client simulates endpoint update → verify dial to new endpoint is attempted
- [ ] Non-HA deployment (no field): zero behavior change — verified by existing test suite passing unchanged
- [ ] Demo: after failover in Kind, worker-1 reconnects to hub-standby without kubectl intervention

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.