canonical / canonical/postgresql-single-kernel-library

planned_units() reads crash config render with ModelError while a force-removed cross-model SAAS lingers in goal-state

Open
#285 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
4
Forks
3
Avg merge
3d 17h
Merged PRs (30d)
26

Description

## Issue

After a dead-DC teardown that force-removes a cross-model SAAS (`juju remove-saas --force`), the removed SAAS application keeps lingering in goal-state, and `Application.planned_units()` fails with `ModelError` for every hook invocation while it does. The library reads `planned_units()` unguarded in two places, so every event that re-renders the Patroni config crashes the hook — and `update-status` re-crashes it on every run for as long as the SAAS lingers (observed for the full 12-minute window in a live repro).

Observed on the PostgreSQL VM charm (`postgresql-operator` revN built on this library, substrate VM/LXD; library pinned at 16.3.6 and verified still present on `16/edge`):

```
ops.model.ModelError: ERROR saas application "db1" not found
_on_async_relation_broken (async_replication.py:559)
-> charm.update_config (charm.py:2774)
-> lib config.py:509 update_config
-> config.py:687 render_patroni_yml_file
-> state.py:505 synchronous_node_count [cached_property]
-> peer_relation.py:414 planned_units
-> ops planned_units -> goal_state() [fails]
```

## Where

`core/state.py` (16.3.6; unchanged on `16/edge`):

```python
@cached_property
def synchronous_node_count(self) -> int:
"""Number of expected sync standbys."""
planned_units = self.application.planned_units # <- raises ModelError
```

`core/peer_relation.py` (both `PostgreSQLPeer` and `PostgreSQLApplication`):

```python
@cached_property
def planned_units(self) -> int:
"""Get the number of planned units for the application."""
return self.app.planned_units() # <- unguarded ops goal-state read
```

`synchronous_node_count` is evaluated unconditionally during `render_patroni_yml_file` (the synchronous block is rendered even when synchronous mode is off), so *any* `update_config()` path is affected — not only replication-related handlers.

## Why a charm-side guard is not enough

The charm (`canonical/postgresql-operator`) added a `_planned_units` fallback for its own direct read (https://github.com/canonical/postgresql-operator/pull/1913), but the library's internal read chain cannot be intercepted from the charm: `charm.update_config()` is called from ~12 sites (update-status, leader-elected, databases-changed, peer-relation events, TLS push, restore completion, password rotation...), and only one of them catches `ModelError` — the others catch `RetryError`/`psycopg2.OperationalError` or nothing, so the lib read crashes those hooks during the lingering-goal-state window.

## Suggested fix

Make the library resilient to transient goal-state failures, e.g.:

- catch `ModelError` in `planned_units` (both `PostgreSQLPeer` and `PostgreSQLApplication`) and fall back to the count of currently known units (`len(self.relation.units)` style), or return `None` and have `synchronous_node_count` skip the synchronous block when the count is unavailable;
- and/or guard `synchronous_node_count`/`synchronous_configuration` so a failed `planned_units` read does not abort the whole config render.

A goal-state outage should degrade the synchronous block, not crash every hook that renders config.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in core/peer_relation.py with planned_units for PostgreSQLPeer and PostgreSQLApplication, then trace synchronous_node_count in core/state.py through render_patroni_yml_file. Compare the two unguarded goal-state reads and the available relation-unit fallback. Done means a lingering SAAS goal-state ModelError no longer aborts configuration rendering or repeated hook invocations.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.