canonical / canonical/charm-integration-testing
test_deploy: mysql-k8s replication handshake race (target blocked, neighbor still setting up)
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 96
Description
**Note**: this is most likely a `mysql-k8s` charm-side bug rather than a test-harness bug. Filing here first since it was surfaced via `test_deploy` triage; if confirmed charm-side, please transfer/re-file against the likely upstream repo `canonical/mysql-k8s-operator` (verify exact repo name).
## Summary
A single observed `test_deploy` failure for `mysql-k8s` revision 426 shows a two-unit replication handshake race: the `target` unit reports `blocked / Ready to create replication` while the `neighbor` unit simultaneously reports `maintenance / Setting up replication` — each side appears to be waiting on the other, and neither converges to `active` within the test's timeout. This is a long-tail signature (n=1 in the last 60 days) but is included here because it is concretely reproducible from a single CI run and represents a distinct failure mode from the other categories.
## Environment
- Charm: `mysql-k8s` revision 426
- Test harness: `charm-integration-testing` `test_deploy` plan
## Evidence
| test_result_id | test_execution_id | CI run |
|---|---|---|
| 12433383 | 778266 | https://github.com/canonical/charm-integration-testing/actions/runs/29805897202 |
Representative status lines:
```
[maintenance] neighbor: Setting up replication
[blocked] target: Ready to create replication
```
## Root cause hypothesis
A handshake/ordering race between the two `mysql-k8s` units during initial replication setup: `target` reaches "ready to create replication" and stops, apparently waiting for `neighbor` to finish its own setup step, while `neighbor` is still mid-setup — if there's a missing signal/relation-data update from `neighbor` back to `target` (or vice versa) once setup completes, both units could stall indefinitely instead of just being transiently slow.
## Reproduction steps
1. Deploy two `mysql-k8s` (rev 426) units via the same `test_deploy` bundle as the CI run above.
2. Watch both units' status simultaneously (`juju status --watch 5s`) through the replication bootstrap sequence.
3. If reproducible, capture the full `io_log`/juju debug-log for `test_execution_id=778266` to see whether the sequence eventually resolves after a longer wait, or truly deadlocks.
4. Given n=1, first confirm this isn't simply a slow-but-eventually-successful convergence — rerun with an extended timeout before concluding it's a genuine deadlock.
## Suggested fix
If confirmed as a genuine stall (not just slow): ensure the unit that finishes "Setting up replication" (`neighbor`) emits/updates whatever relation data or event the `target` unit (`Ready to create replication`) is waiting on to proceed.
## Suggested regression test
Add a scenario/integration test that deploys two `mysql-k8s` units fresh and asserts both reach `active` within a bounded, generous timeout, capturing the intermediate `blocked`/`maintenance` handshake states — this would need more recurrences to build a scenario, so consider first watching for additional occurrences in future untriaged batches before investing heavily in this one.
Suggested labels: `bug`, `test_deploy`, `mysql-k8s`
Contributor guide
Research direction
Start with the test_deploy plan and rerun the same two-unit mysql-k8s revision 426 deployment with an extended timeout. Watch `juju status --watch 5s` and collect the full io_log and juju debug-log for test_execution_id=778266; compare whether the blocked/maintenance state resolves or deadlocks. Done means confirming the race and identifying the missing relation update or event, or documenting that it is only slow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, mysql, python
- Domain
- devops, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100