canonical / canonical/postgresql-k8s-operator

[postgresql-k8s] config-changed hook fails permanently instead of deferring when DB is unavailable post-migration

Open
#1,406 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
15
Forks
33
Avg merge
1d 3h
Merged PRs (30d)
41

Description

> ⚠️ This issue was generated with AI assistance (GitHub Copilot) as part of automated test triage.

## Summary
`postgresql-k8s` (rev 845, channel `14/edge`) fails its `config-changed` hook after Juju model controller migration. When the PostgreSQL container restarts post-migration, `_handle_enable_disable_extensions` attempts a psycopg2 connection that fails with `OperationalError: Operation not permitted`. Instead of deferring the event, the charm raises `PostgreSQLEnableDisableExtensionError` and permanently fails the hook, leaving the unit in error state.

## Observed Failure
- **Test**: `test_model_controller_migration`
- **Test Execution**: 451797
- **Test Result ID**: 10243108
- **Test Observer**: https://test-observer.canonical.com/#/charms/406319?testExecutionId=451797&testResultId=10243108
- **CI Run**: https://github.com/canonical/charm-integration-testing/actions/runs/23900665268

## Error
```
charm_qa:failure:charm:postgresql-k8s:status: unit:error:hook failed: "config-changed"
```

```
juju.backend.JujuWaitTimeoutError: Timed out while waiting for applications: [active],
units: [active] (units: ['target/0'])
```

## Traceback (from concurrent execution with identical charm version and failure signature)
```python
# charm.py:754 in _handle_enable_disable_extensions
self.postgresql.enable_disable_extensions(extensions, database)

# postgresql.py:442 in enable_disable_extensions
with self._connect_to_database() as connection, connection.cursor() as cursor:

# postgresql.py:184 in _connect_to_database
connection = psycopg2.connect(...)

# FAILS:
psycopg2.OperationalError: connection to server at
"target-primary.model-XXXX.svc.cluster.local" (10.x.x.x), port 5432 failed:
Operation not permitted
→ raises PostgreSQLEnableDisableExtensionError
```

## Failure Sequence

**Step 1**: Model migrated to temporary controller `pytest-tmp-controller-879t6qlq` and back. `test_model_controller_migration` validators all pass, test reports PASSED.

**Step 2**: Post-migration, the PostgreSQL container restarts (`container "postgresql": pebbleNoticer stopped` observed multiple times in logs).

**Step 3**: `config-changed` hook fires while the container is still restarting. `_handle_enable_disable_extensions` attempts to connect to PostgreSQL via psycopg2.

**Step 4**: Connection fails with `OperationalError: Operation not permitted` (K8s network not yet ready). The charm raises `PostgreSQLEnableDisableExtensionError` and **fails the hook permanently** rather than deferring.

**Step 5**: Unit enters `error: hook failed: "config-changed"` and never recovers.

## Secondary Issue
A K8s resource conflict was observed on the `_on_stop` hook:
```
lightkube.core.exceptions.ApiError: Operation cannot be fulfilled on services
"patroni-target-config": the object has been modified; please apply your changes
to the latest version and try again
(HTTP 409 Conflict: https://.../services/patroni-target-config?force=true)
```
This suggests a concurrency race when patching the Patroni K8s service during teardown.

## Suggested Fix
In `_handle_enable_disable_extensions`, when `PostgreSQLEnableDisableExtensionError` is raised due to a connection failure, the charm should defer the event (or set Waiting status and retry) rather than permanently failing the hook:

```python
# charm.py — in _handle_enable_disable_extensions
try:
self.postgresql.enable_disable_extensions(extensions, database)
except PostgreSQLEnableDisableExtensionError as e:
if "Operation not permitted" in str(e) or "connection" in str(e).lower():
logger.warning("DB not yet available, deferring config-changed")
event.defer() # ← defer instead of failing
return
raise
```

## Failure History (not a regression — persistent issue across revisions)
| Execution | Revision | Result |
|---|---|---|
| 451797 | 845 | ❌ FAILED |
| 450782 | 843 | ❌ FAILED |
| 450781 | 843 | ❌ FAILED |
| 450783 | 843 | ❌ FAILED |
| 450780 | 843 | ✅ PASSED |
| 450779 | 843 | ✅ PASSED |

~30–50% failure rate across revisions 843 and 845.

## Charms in Bundle
- `postgresql-k8s` rev 845 (14/edge) ← **failing charm**
- `glauth-k8s` rev 57 (neighbor)
- `self-signed-certificates` rev 586

## Environment
- Juju: 3.6.20
- Cloud: k8s-production (kubernetes)
- Ubuntu: 22.04
- Date: 2026-04-02

Contributor guide

Open the contributing guide

Research direction

Start in charm.py at _handle_enable_disable_extensions, then trace the connection path through postgresql.py’s enable_disable_extensions and _connect_to_database. Run test_model_controller_migration and inspect the reported post-migration failure. Done means a temporarily unavailable database does not permanently fail config-changed and the unit recovers; the separate patroni-target-config conflict may need independent investigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql, python
Domain
backend, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.