kubeslice / kubeslice/worker-operator

tests: failover robustness tests (reconnect + trust validation)

Open
#468 0 comments 0 reactions 1 assignee Claimed by @gourishkb View on GitHub
enhancement feature lfx-mentorship
Dominant language
Go
Stars
62
Forks
33
Avg merge
3d 10h
Merged PRs (30d)
3

Description

## Summary

Add automated tests validating worker operator behavior across a full controller switchover: disconnect, reconnect, cert trust validation, and confirmation that the traffic plane is unaffected.

**Depends on:** #467 (worker reconnect implemented), kubeslice-controller #297 (promotion logic complete)

---

## Test scenarios

### Scenario 1: Clean failover — worker reconnects
```
Setup: hub-active running, hub-standby running, worker-1 connected to hub-active
1. Kill hub-active controller
2. Wait for hub-standby to promote (via ha_failover_total metric or Event)
3. Assert: worker-1 receives updated controllerEndpoint in Cluster CR
4. Assert: worker-1 operator dials hub-standby within --ha-reconnect-timeout
5. Assert: ControllerConnected condition = True on worker-1
6. Assert: WorkerSliceConfig reconciliation resumes on worker-1 (create a new Slice post-failover)
```

### Scenario 2: Cert trust validation
```
1. Perform failover (Scenario 1 steps 1–3)
2. Inject an invalid CA bundle into the updated controllerCABundle field
3. Assert: worker-1 logs TLS handshake failure, does NOT establish connection
4. Assert: ControllerConnected condition = False, reason=CertVerificationFailed
5. Fix CA bundle to valid value
6. Assert: worker-1 reconnects successfully
```

### Scenario 3: Traffic plane unaffected during failover
```
1. Establish active data-plane traffic between two worker clusters (ping or iperf)
2. Trigger failover (kill hub-active)
3. Assert: data-plane traffic continues uninterrupted during the failover window
4. Assert: no VPN tunnel flap on either worker during the management-plane reconnect
```

### Scenario 4: Non-HA worker (backward compat)
```
1. Worker operator running without controllerEndpoint field in Cluster CR
2. Simulate hub restart
3. Assert: worker operator behaves identically to pre-feature behavior
4. Assert: no new errors in worker operator logs attributable to this feature
```

---

## CI requirements

- Tests runnable with `make test-e2e-ha` using Kind clusters
- Must share Kind cluster setup with kubeslice-controller #299 if possible (joint test suite)
- Each scenario must complete within 120s
- Scenario 3 requires a mechanism to generate and measure data-plane traffic in Kind (document the approach used)

---

## Acceptance Criteria

- [ ] All four scenarios above are implemented and pass
- [ ] Scenario 2 (cert validation) is covered — this is a security-critical path
- [ ] Scenario 3 (traffic plane) explicitly validates zero tunnel downtime during failover
- [ ] CI runnable without manual intervention

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.