canonical / canonical/charm-integration-testing
test_deploy: pgbouncer-k8s stuck `blocked` on backend-database relation
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
**Note**: this is most likely a `pgbouncer-k8s` charm-side bug rather than a test-harness bug (or possibly the `test_deploy` timeout is simply too short for this charm combination). Filing here first since it was surfaced via `test_deploy` triage; if confirmed charm-side, please transfer/re-file against the likely upstream repo `canonical/pgbouncer-k8s-operator` (verify exact repo name).
## Summary
In `test_deploy` integration test runs, `pgbouncer-k8s` frequently never leaves the `blocked` status `waiting for backend database relation to initialise`, causing the overall test to time out waiting for all applications to reach `active`. This is the single largest contributor to `blocked` failures in `test_deploy`: **25 of 62** untriaged `blocked` failures over the last 60 days show this exact message (16 as the primary `pgbouncer-k8s` artefact, 9 as a `pgbouncer-k8s` neighbor of `openfga-k8s`).
## Environment
- Charms observed: `pgbouncer-k8s` rev 562 (as target and as neighbor), `openfga-k8s` rev 123
- Companion charm: `postgresql-k8s`
- Juju version: 3.6.x (per collected `juju-status.txt` headers)
- Test harness: `charm-integration-testing` `test_deploy` plan
## Evidence (reproducible CI runs)
| test_result_id | test_execution_id | artefact | CI run |
|---|---|---|---|
| 12384969 | 755870 | sloth-k8s 19 (pgbouncer-k8s neighbor) | https://github.com/canonical/charm-integration-testing/actions/runs/29162347175 |
| 12383466 | 755827 | vault-k8s 354 (pgbouncer-k8s neighbor) | https://github.com/canonical/charm-integration-testing/actions/runs/29157798063 |
| 12382080 | 752953 | openfga-k8s 123 | https://github.com/canonical/charm-integration-testing/actions/runs/29152437895 |
| 12382044 | 752833 | openfga-k8s 123 | https://github.com/canonical/charm-integration-testing/actions/runs/29152211913 |
Representative status line:
```
[blocked] pgbouncer-k8s: waiting for backend database relation to initialise
```
## Root cause hypothesis
`pgbouncer-k8s`'s `backend-database` relation to `postgresql-k8s` does not settle within the test's wait window. Either:
1. `pgbouncer-k8s`'s relation-changed/relation-joined handler for `backend-database` isn't reacting promptly to `postgresql-k8s` becoming ready (a charm-side bug), or
2. `postgresql-k8s` itself is slow to publish the relation data `pgbouncer-k8s` waits on, and the `test_deploy` timeout is simply too short for this combination (a test-harness tuning issue, not a charm bug).
## Reproduction steps
1. Deploy `pgbouncer-k8s` rev 562 (or `openfga-k8s` rev 123, which pulls in `pgbouncer-k8s` as a dependency) together with `postgresql-k8s` using the same `test_deploy` bundle as the CI runs above.
2. `juju status --watch 5s` and observe `pgbouncer-k8s` remaining at `blocked / waiting for backend database relation to initialise` past the test's timeout, while `postgresql-k8s` itself reaches `active`.
3. Compare relation data on both sides of `backend-database` (`juju show-unit`, `relation-get`) to see whether `pgbouncer-k8s` is missing expected keys or simply not re-evaluating on relation-changed.
## Suggested fix
- If charm-side: ensure `pgbouncer-k8s`'s relation-changed observer for `backend-database` retries or watches for eventual consistency (e.g. via `update-status` or deferred events) rather than giving up after the first evaluation.
- If test-harness-side: increase the `test_deploy` wait timeout for bundles that include `pgbouncer-k8s` + `postgresql-k8s`, since the settle time may simply exceed the current bound.
## Suggested regression test
Add an integration test that deploys `pgbouncer-k8s` + `postgresql-k8s` fresh and asserts `pgbouncer-k8s` reaches `active` within a fixed, generous timeout, failing fast with the relation data dumped if it doesn't.
Suggested labels: `bug`, `test_deploy`, `pgbouncer-k8s`
Contributor guide
Research direction
Start with the charm-integration-testing test_deploy plan and the linked CI runs, then reproduce the deployment with pgbouncer-k8s and postgresql-k8s. Use juju status, juju show-unit, and relation-get to inspect the backend-database relation and determine whether the relation data is missing or the status is not re-evaluated. Done means the cause is identified and the deployment reaches active within an appropriate timeout, with a regression test covering the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases, devops, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100