canonical / canonical/charm-integration-testing
test_deploy: mongodb-k8s neighbor postgresql-k8s crashes with hook failed "leader-elected"
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 96
Description
**Note**: this is most likely a `postgresql-k8s` charm-side hook crash rather than a test-harness bug. Filing here first since it was surfaced via `test_deploy` triage; if confirmed charm-side, please transfer/re-file against the likely upstream repo `canonical/postgresql-k8s-operator` (verify exact repo name).
## Summary
Within the `mongodb-k8s` (revision 205) "all apps active" timeout bucket (13 of 101 total), a distinct sub-signature shows the `postgresql-k8s` neighbor unit crashing with `hook failed: "leader-elected"` (`workload=error`, `agent=idle`), rather than merely being slow to converge. This is a genuine charm hook crash, separate from the other `mongodb-k8s` sub-pattern where units are simply still mid-`config-changed` (which is likely just slow convergence, not a bug).
## Environment
- Charm under test: `mongodb-k8s` revision 205
- Neighbor charm: `postgresql-k8s`
- Test harness: `charm-integration-testing` `test_deploy` plan
## Evidence
Executions showing the crash (test_execution_id, from the collected `juju-status.txt` unit tables):
- `exec=620009` — `postgresql-k8s/0* workload=error agent=idle message='hook failed: "leader-elected"'`
- `exec=620005` — same signature
(Use `search_test_results`/`get_test_execution` with these `test_execution_id`s against the Test Observer API to resolve the corresponding `test_result_id` and CI run link.)
Contrast — a **different**, likely benign sub-pattern also present in this artefact's "all-active" bucket (units still converging, not crashed):
- `exec=620045` — `grafana-k8s/0* workload=active agent=executing message="(config-changed)"`
- `exec=620039` — `neighbor/0* workload=active agent=executing message="(config-changed)"`
- `exec=620035` — `grafana-k8s/0* workload=active agent=executing message="(config-changed)"`
## Root cause hypothesis
`postgresql-k8s`'s `leader-elected` hook is crashing (uncaught exception or non-zero exit) under some condition triggered during `mongodb-k8s`'s `test_deploy` scenario — possibly a race where leadership is (re-)granted before some prerequisite state (e.g. peer relation data, storage mount) is ready.
## Reproduction steps
1. Deploy `mongodb-k8s` revision 205 together with `postgresql-k8s` via the same `test_deploy` bundle.
2. Force or wait for a leader-elected event on `postgresql-k8s` (e.g. scale down to 0 units then back up to 1, or kill the current leader unit) and watch for the hook to crash.
3. Pull the full `io_log`/juju debug-log for `test_execution_id=620009` to get the full Python traceback behind the `hook failed: "leader-elected"` crash.
## Suggested fix
Once the traceback is retrieved, fix the underlying exception in `postgresql-k8s`'s `leader-elected` observer (likely a missing null-check or a relation-data race).
## Suggested regression test
Add a `postgresql-k8s` unit/scenario test that fires a `leader-elected` event before some dependent piece of state exists (mirroring the race above) and asserts the hook completes without raising.
Suggested labels: `bug`, `test_deploy`, `mongodb-k8s`
Contributor guide
Research direction
Start with the test_deploy plan and use search_test_results/get_test_execution for executions 620009 and 620005. Retrieve the full io_log or Juju debug log to identify the traceback behind the postgresql-k8s leader-elected failure, then verify the exact upstream repository. Done means the underlying crash is fixed there and a regression test covers the event before dependent state exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100