canonical / canonical/charm-integration-testing

test_deploy: pgbouncer-k8s stuck `blocked` on backend-database relation

Open
#794 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 20h
Merged PRs (30d)
98

Description

**Note**: this is most likely a `pgbouncer-k8s` charm-side bug rather than a test-harness bug (or possibly the `test_deploy` timeout is simply too short for this charm combination). Filing here first since it was surfaced via `test_deploy` triage; if confirmed charm-side, please transfer/re-file against the likely upstream repo `canonical/pgbouncer-k8s-operator` (verify exact repo name).

## Summary

In `test_deploy` integration test runs, `pgbouncer-k8s` frequently never leaves the `blocked` status `waiting for backend database relation to initialise`, causing the overall test to time out waiting for all applications to reach `active`. This is the single largest contributor to `blocked` failures in `test_deploy`: **25 of 62** untriaged `blocked` failures over the last 60 days show this exact message (16 as the primary `pgbouncer-k8s` artefact, 9 as a `pgbouncer-k8s` neighbor of `openfga-k8s`).

## Environment

- Charms observed: `pgbouncer-k8s` rev 562 (as target and as neighbor), `openfga-k8s` rev 123
- Companion charm: `postgresql-k8s`
- Juju version: 3.6.x (per collected `juju-status.txt` headers)
- Test harness: `charm-integration-testing` `test_deploy` plan

## Evidence (reproducible CI runs)

| test_result_id | test_execution_id | artefact | CI run |
|---|---|---|---|
| 12384969 | 755870 | sloth-k8s 19 (pgbouncer-k8s neighbor) | https://github.com/canonical/charm-integration-testing/actions/runs/29162347175 |
| 12383466 | 755827 | vault-k8s 354 (pgbouncer-k8s neighbor) | https://github.com/canonical/charm-integration-testing/actions/runs/29157798063 |
| 12382080 | 752953 | openfga-k8s 123 | https://github.com/canonical/charm-integration-testing/actions/runs/29152437895 |
| 12382044 | 752833 | openfga-k8s 123 | https://github.com/canonical/charm-integration-testing/actions/runs/29152211913 |

Representative status line:
```
[blocked] pgbouncer-k8s: waiting for backend database relation to initialise
```

## Root cause hypothesis

`pgbouncer-k8s`'s `backend-database` relation to `postgresql-k8s` does not settle within the test's wait window. Either:
1. `pgbouncer-k8s`'s relation-changed/relation-joined handler for `backend-database` isn't reacting promptly to `postgresql-k8s` becoming ready (a charm-side bug), or
2. `postgresql-k8s` itself is slow to publish the relation data `pgbouncer-k8s` waits on, and the `test_deploy` timeout is simply too short for this combination (a test-harness tuning issue, not a charm bug).

## Reproduction steps

1. Deploy `pgbouncer-k8s` rev 562 (or `openfga-k8s` rev 123, which pulls in `pgbouncer-k8s` as a dependency) together with `postgresql-k8s` using the same `test_deploy` bundle as the CI runs above.
2. `juju status --watch 5s` and observe `pgbouncer-k8s` remaining at `blocked / waiting for backend database relation to initialise` past the test's timeout, while `postgresql-k8s` itself reaches `active`.
3. Compare relation data on both sides of `backend-database` (`juju show-unit`, `relation-get`) to see whether `pgbouncer-k8s` is missing expected keys or simply not re-evaluating on relation-changed.

## Suggested fix

- If charm-side: ensure `pgbouncer-k8s`'s relation-changed observer for `backend-database` retries or watches for eventual consistency (e.g. via `update-status` or deferred events) rather than giving up after the first evaluation.
- If test-harness-side: increase the `test_deploy` wait timeout for bundles that include `pgbouncer-k8s` + `postgresql-k8s`, since the settle time may simply exceed the current bound.

## Suggested regression test

Add an integration test that deploys `pgbouncer-k8s` + `postgresql-k8s` fresh and asserts `pgbouncer-k8s` reaches `active` within a fixed, generous timeout, failing fast with the relation data dumped if it doesn't.

Suggested labels: `bug`, `test_deploy`, `pgbouncer-k8s`

Contributor guide

Open the contributing guide

Research direction

Start with the charm-integration-testing test_deploy plan and the linked CI runs, then reproduce the deployment with pgbouncer-k8s and postgresql-k8s. Use juju status, juju show-unit, and relation-get to inspect the backend-database relation and determine whether the relation data is missing or the status is not re-evaluated. Done means the cause is identified and the deployment reaches active within an appropriate timeout, with a regression test covering the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.