canonical / canonical/livepatch-k8s-operator

database-relation-changed hook fails with SQLSTATE 42P05 (prepared statement "lrupsc_1_0" already exists) when pgbouncer-k8s is redeployed

Open
#96 0 comments 1 reaction 1 assignee Claimed by @trollLemon View on GitHub
Dominant language
Python
Stars
4
Forks
12
Avg merge
13h 21m
Merged PRs (30d)
4

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

---

## Summary

The `database-relation-changed` hook in `canonical-livepatch-server-k8s` rev 76 raises an uncaught exception when `pgbouncer-k8s` (the database proxy) is redeployed or scaled. The `livepatch-schema-tool check` command fails with PostgreSQL error `SQLSTATE 42P05: prepared statement "lrupsc_1_0" already exists`, causing the unit to enter error state.

## Test Observer

- **Execution**: https://test-observer.canonical.com/#/charms/405104?testExecutionId=413592&testResultId=9533915
- **Test plan**: `integration/postgresql-k8s:database/postgresql_client/pgbouncer-k8s:backend-database`
- **CI run**: https://github.com/canonical/charm-integration-testing/actions/runs/22491507153
- **Charm**: `canonical-livepatch-server-k8s` rev 76 (`latest/stable`), pgbouncer-k8s rev 520 (`1/edge`)

## Error (from juju debug-log)

Observed at `15:34:19` during idempotent redeploy of pgbouncer-k8s, and again at `15:20:17` during scale-in/scale-out:

```
ERROR unit.canonical-livepatch-server-k8s/0.juju-log database:11:
Failed to determine if schema upgrade required: non-zero exit code 1 executing
['/usr/local/bin/livepatch-schema-tool', 'check', '/etc/livepatch/schema-upgrades',
'--db', 'postgresql://[REDACTED]@target-service.../livepatch-server'],
stderr='Error: ERROR: prepared statement "lrupsc_1_0" already exists (SQLSTATE 42P05)
ERROR: prepared statement "lrupsc_1_0" already exists (SQLSTATE 42P05)'

ERROR unit.canonical-livepatch-server-k8s/0.juju-log database:11:
Uncaught exception while in charm code:

ERROR juju.worker.uniter.operation hook "database-relation-changed"
(via hook dispatching script: dispatch) failed: exit status 1
```

## Root Cause

When pgbouncer-k8s is redeployed or scaled, a `database-relation-changed` event fires. The charm calls `livepatch-schema-tool check` which internally prepares a PostgreSQL statement named `lrupsc_1_0`. In pgBouncer **session pooling** mode, the backend PostgreSQL connection may be reused across multiple invocations, meaning a previously prepared statement `lrupsc_1_0` still exists on the server-side connection. The subsequent `PREPARE` then fails with `42P05`.

The charm has no error handling for this exit code from `livepatch-schema-tool`, resulting in an uncaught exception and a hook failure.

## Recurrence

This failure is reproducible across multiple test executions for pgbouncer-k8s rev 520:
| Execution | Result |
|-----------|--------|
| 413592 | FAILED |
| 413565 | FAILED |
| 413563 | FAILED |
| 413589 | FAILED |

## Suggested Fix

One or more of the following:
1. **In `livepatch-schema-tool`**: Execute `DEALLOCATE lrupsc_1_0` (or `DEALLOCATE ALL`) before re-preparing the statement, so it is idempotent.
2. **In `livepatch-schema-tool`**: Treat `SQLSTATE 42P05` as non-fatal — if the statement already exists, it can safely be reused or re-prepared after deallocation.
3. **In the charm's `database-relation-changed` handler**: Catch the exit-code-1 error from `livepatch-schema-tool` and handle `42P05` gracefully instead of propagating as uncaught exception.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.