kubeslice / kubeslice/worker-operator
tests: failover robustness tests (reconnect + trust validation)
- Dominant language
- Go
- Stars
- 62
- Forks
- 33
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 3
Description
## Summary
Add automated tests validating worker operator behavior across a full controller switchover: disconnect, reconnect, cert trust validation, and confirmation that the traffic plane is unaffected.
**Depends on:** #467 (worker reconnect implemented), kubeslice-controller #297 (promotion logic complete)
---
## Test scenarios
### Scenario 1: Clean failover — worker reconnects
```
Setup: hub-active running, hub-standby running, worker-1 connected to hub-active
1. Kill hub-active controller
2. Wait for hub-standby to promote (via ha_failover_total metric or Event)
3. Assert: worker-1 receives updated controllerEndpoint in Cluster CR
4. Assert: worker-1 operator dials hub-standby within --ha-reconnect-timeout
5. Assert: ControllerConnected condition = True on worker-1
6. Assert: WorkerSliceConfig reconciliation resumes on worker-1 (create a new Slice post-failover)
```
### Scenario 2: Cert trust validation
```
1. Perform failover (Scenario 1 steps 1–3)
2. Inject an invalid CA bundle into the updated controllerCABundle field
3. Assert: worker-1 logs TLS handshake failure, does NOT establish connection
4. Assert: ControllerConnected condition = False, reason=CertVerificationFailed
5. Fix CA bundle to valid value
6. Assert: worker-1 reconnects successfully
```
### Scenario 3: Traffic plane unaffected during failover
```
1. Establish active data-plane traffic between two worker clusters (ping or iperf)
2. Trigger failover (kill hub-active)
3. Assert: data-plane traffic continues uninterrupted during the failover window
4. Assert: no VPN tunnel flap on either worker during the management-plane reconnect
```
### Scenario 4: Non-HA worker (backward compat)
```
1. Worker operator running without controllerEndpoint field in Cluster CR
2. Simulate hub restart
3. Assert: worker operator behaves identically to pre-feature behavior
4. Assert: no new errors in worker operator logs attributable to this feature
```
---
## CI requirements
- Tests runnable with `make test-e2e-ha` using Kind clusters
- Must share Kind cluster setup with kubeslice-controller #299 if possible (joint test suite)
- Each scenario must complete within 120s
- Scenario 3 requires a mechanism to generate and measure data-plane traffic in Kind (document the approach used)
---
## Acceptance Criteria
- [ ] All four scenarios above are implemented and pass
- [ ] Scenario 2 (cert validation) is covered — this is a security-critical path
- [ ] Scenario 3 (traffic plane) explicitly validates zero tunnel downtime during failover
- [ ] CI runnable without manual intervention
Contributor guide
Assessment
This issue has not been assessed yet.