canonical / canonical/postgresql-k8s-operator
[postgresql-k8s] config-changed hook fails permanently instead of deferring when DB is unavailable post-migration
- Dominant language
- Python
- Stars
- 15
- Forks
- 33
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 41
Description
> ⚠️ This issue was generated with AI assistance (GitHub Copilot) as part of automated test triage.
## Summary
`postgresql-k8s` (rev 845, channel `14/edge`) fails its `config-changed` hook after Juju model controller migration. When the PostgreSQL container restarts post-migration, `_handle_enable_disable_extensions` attempts a psycopg2 connection that fails with `OperationalError: Operation not permitted`. Instead of deferring the event, the charm raises `PostgreSQLEnableDisableExtensionError` and permanently fails the hook, leaving the unit in error state.
## Observed Failure
- **Test**: `test_model_controller_migration`
- **Test Execution**: 451797
- **Test Result ID**: 10243108
- **Test Observer**: https://test-observer.canonical.com/#/charms/406319?testExecutionId=451797&testResultId=10243108
- **CI Run**: https://github.com/canonical/charm-integration-testing/actions/runs/23900665268
## Error
```
charm_qa:failure:charm:postgresql-k8s:status: unit:error:hook failed: "config-changed"
```
```
juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active],
units: [active] (units: ['target/0'])
```
## Traceback (from concurrent execution with identical charm version and failure signature)
```python
# charm.py:754 in _handle_enable_disable_extensions
self.postgresql.enable_disable_extensions(extensions, database)
# postgresql.py:442 in enable_disable_extensions
with self._connect_to_database() as connection, connection.cursor() as cursor:
# postgresql.py:184 in _connect_to_database
connection = psycopg2.connect(...)
# FAILS:
psycopg2.OperationalError: connection to server at
"target-primary.model-XXXX.svc.cluster.local" (10.x.x.x), port 5432 failed:
Operation not permitted
→ raises PostgreSQLEnableDisableExtensionError
```
## Failure Sequence
**Step 1**: Model migrated to temporary controller `pytest-tmp-controller-879t6qlq` and back. `test_model_controller_migration` validators all pass, test reports PASSED.
**Step 2**: Post-migration, the PostgreSQL container restarts (`container "postgresql": pebbleNoticer stopped` observed multiple times in logs).
**Step 3**: `config-changed` hook fires while the container is still restarting. `_handle_enable_disable_extensions` attempts to connect to PostgreSQL via psycopg2.
**Step 4**: Connection fails with `OperationalError: Operation not permitted` (K8s network not yet ready). The charm raises `PostgreSQLEnableDisableExtensionError` and **fails the hook permanently** rather than deferring.
**Step 5**: Unit enters `error: hook failed: "config-changed"` and never recovers.
## Secondary Issue
A K8s resource conflict was observed on the `_on_stop` hook:
```
lightkube.core.exceptions.ApiError: Operation cannot be fulfilled on services
"patroni-target-config": the object has been modified; please apply your changes
to the latest version and try again
(HTTP 409 Conflict: https://.../services/patroni-target-config?force=true)
```
This suggests a concurrency race when patching the Patroni K8s service during teardown.
## Suggested Fix
In `_handle_enable_disable_extensions`, when `PostgreSQLEnableDisableExtensionError` is raised due to a connection failure, the charm should defer the event (or set Waiting status and retry) rather than permanently failing the hook:
```python
# charm.py — in _handle_enable_disable_extensions
try:
self.postgresql.enable_disable_extensions(extensions, database)
except PostgreSQLEnableDisableExtensionError as e:
if "Operation not permitted" in str(e) or "connection" in str(e).lower():
logger.warning("DB not yet available, deferring config-changed")
event.defer() # ← defer instead of failing
return
raise
```
## Failure History (not a regression — persistent issue across revisions)
| Execution | Revision | Result |
|---|---|---|
| 451797 | 845 | ❌ FAILED |
| 450782 | 843 | ❌ FAILED |
| 450781 | 843 | ❌ FAILED |
| 450783 | 843 | ❌ FAILED |
| 450780 | 843 | ✅ PASSED |
| 450779 | 843 | ✅ PASSED |
~30–50% failure rate across revisions 843 and 845.
## Charms in Bundle
- `postgresql-k8s` rev 845 (14/edge) ← **failing charm**
- `glauth-k8s` rev 57 (neighbor)
- `self-signed-certificates` rev 586
## Environment
- Juju: 3.6.20
- Cloud: k8s-production (kubernetes)
- Ubuntu: 22.04
- Date: 2026-04-02
Contributor guide
Research direction
Start in charm.py at _handle_enable_disable_extensions, then trace the connection path through postgresql.py’s enable_disable_extensions and _connect_to_database. Run test_model_controller_migration and inspect the reported post-migration failure. Done means a temporarily unavailable database does not permanently fail config-changed and the unit recovers; the separate patroni-target-config conflict may need independent investigation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, postgresql, python
- Domain
- backend, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100