Replica units stuck in permanent bootstrap loop on MicroCloud (EBUSY on Juju storage mount)

## Steps to reproduce

Prerequisites: A host machine running LXD 5.21+ with at least 30GB free disk space.

### MicroCloud setup

```bash
# 1. Create 3 containers with nesting enabled and privileged mode for MicroCloud
# Note: security.privileged=true is required on the outer containers so that
# doubly-nested containers (created by Juju inside mc1's LXD) can use snaps.
# This is due to a known AppArmor limitation where AppArmor namespaces only
# support a single level of nesting (https://github.com/canonical/lxd/issues/14770).
lxc launch ubuntu:24.04 mc1 -c security.nesting=true -c security.privileged=true
lxc launch ubuntu:24.04 mc2 -c security.nesting=true -c security.privileged=true
lxc launch ubuntu:24.04 mc3 -c security.nesting=true -c security.privileged=true

# 2. Wait for containers to start, then install MicroCloud snaps on all 3
# (MicroCeph is not required for this reproduction)
# Note: Install LXD on mc1 first, then use the same revision on mc2/mc3
# to ensure all cluster members run the same version (required by MicroCloud).
lxc exec mc1 -- bash -c '
snap install microcloud --channel 2/stable
snap install lxd --channel latest/stable
snap install juju --channel 3.6/stable
'
LXD_REV=$(lxc exec mc1 -- snap list lxd | awk '/^lxd/{print $3}')
for node in mc2 mc3; do
lxc exec $node -- bash -c "
snap install microcloud --channel 2/stable
snap install lxd --revision=${LXD_REV}
snap install juju --channel 3.6/stable
"
done

# 3. Initialize MicroCloud using preseed (non-interactive)
# First, find the subnet used by mc1/mc2/mc3:
SUBNET=$(lxc exec mc1 -- ip -4 addr show eth0 | grep -oP 'inet \K[0-9.]+' | sed 's/\.[0-9]*$/.0\/24/')

# Create preseed config and run on ALL 3 nodes simultaneously:
for node in mc1 mc2 mc3; do
lxc exec $node -- bash -c "cat > /tmp/preseed.yaml << EOF
lookup_subnet: ${SUBNET}
initiator: mc1
session_passphrase: test1234
systems:
- name: mc1
- name: mc2
- name: mc3
EOF"
done

# Run preseed on all nodes at the same time (they must discover each other):
for node in mc1 mc2 mc3; do
lxc exec $node -- bash -c 'cat /tmp/preseed.yaml | microcloud preseed' &
done
wait
# Verify: lxc exec mc1 -- lxc cluster list

# 4. Configure the default profile inside mc1's LXD for doubly-nested containers
# The raw.lxc setting disables AppArmor confinement for inner containers,
# which is needed because AppArmor can't nest beyond one level.
lxc exec mc1 -- bash -c '
lxc profile set default raw.lxc "lxc.apparmor.profile=unconfined"
lxc profile set default security.nesting true
lxc profile set default security.privileged false
lxc profile device add default root disk path=/ pool=local 2>/dev/null \
|| lxc profile device set default root pool=local
lxc profile device add default eth0 nic network=lxdfan0 2>/dev/null \
|| lxc profile device set default eth0 network=lxdfan0
'

# 5. Set up NAT on mc1 for internet access from fan network containers
# (The fan subnet is 240..0.0/16)
MC1_IP=$(lxc exec mc1 -- ip -4 addr show eth0 | grep -oP 'inet \K[0-9.]+')
MC1_LAST_OCTET=$(echo $MC1_IP | awk -F. '{print $4}')
lxc exec mc1 -- iptables -t nat -A POSTROUTING \
-s 240.${MC1_LAST_OCTET}.0.0/16 -o eth0 -j MASQUERADE

# 6. Create storage pools across all cluster members
# MicroCloud may not auto-create 'local' or 'default' pools depending on
# version/config. Create both if they don't already exist.
lxc exec mc1 -- bash -c '
for pool in local default; do
lxc storage create $pool dir --target mc1 2>/dev/null
lxc storage create $pool dir --target mc2 2>/dev/null
lxc storage create $pool dir --target mc3 2>/dev/null
lxc storage create $pool dir 2>/dev/null
done
lxc storage list
'

# 7. Create Juju home directory and bootstrap Juju inside the MicroCloud cluster
lxc exec mc1 -- mkdir -p /root/.local/share/juju
lxc exec mc1 -- juju bootstrap localhost

# 8. Create a model
lxc exec mc1 -- juju add-model dev
```

### Bug reproduction

All commands below are run inside mc1 (`lxc exec mc1 -- bash`):

```bash
# 1. Create a Juju storage pool pointing to the local dir pool
juju create-storage-pool mc-local lxd lxd-pool=local

# 2. Set update-status-hook-interval to avoid interference
juju model-config update-status-hook-interval=60m

# 3. Deploy a single-unit PostgreSQL cluster pinned to one node
juju deploy postgresql --channel 16/stable -n 1 \
--storage data=mc-local,1G \
--storage logs=mc-local,1G \
--storage archive=mc-local,1G \
--storage temp=mc-local,1G \
--constraints zones=mc1

# 4. Wait for the primary to become active
juju status --watch 5s
# Wait until: postgresql/0* active idle ... Primary

# 5. (Optional but increases reliability) Insert data to make pg_basebackup slower
LEADER=$(juju status --format json | python3 -c "import sys,json; d=json.load(sys.stdin); units=d['applications']['postgresql']['units']; print([k for k,v in units.items() if 'Primary' in v.get('workload-status',{}).get('message','')][0])")
PASSWORD=$(juju exec -u $LEADER -- bash -c 'grep -A2 "superuser:" /var/snap/charmed-postgresql/current/etc/patroni/patroni.yaml | grep password | awk "{print \$2}"')
juju exec -u $LEADER -- bash -c "PGPASSWORD=${PASSWORD} charmed-postgresql.psql -h /tmp -U operator -d postgres -c \"CREATE TABLE large_data (id SERIAL PRIMARY KEY, data TEXT); INSERT INTO large_data (data) SELECT repeat(md5(random()::text), 200) FROM generate_series(1, 500000);\""

# 6. Pin constraints and add 2 units
juju set-constraints postgresql zones=mc1
juju add-unit postgresql -n 2

# 7. Observe the stuck state
juju status --watch 5s
# New units will show:
# "awaiting for member to start" or "waiting to start PostgreSQL"
# and will never progress to active.
```

To verify the stuck state, check Patroni logs on the stuck replica unit:

```bash
juju exec -u postgresql/ -- bash -c \
'for f in $(ls -tr /var/snap/charmed-postgresql/common/var/log/patroni/); do
cat /var/snap/charmed-postgresql/common/var/log/patroni/$f
done'
```

You will see a repeating loop of:
```
ERROR: Error when fetching backup: pg_basebackup exited with code=1
WARNING: Trying again in 5 seconds
...
ERROR: Could not remove data directory /var/snap/charmed-postgresql/common/var/lib/postgresql
OSError: [Errno 16] Device or resource busy
...
ERROR: Could not rename data directory /var/snap/charmed-postgresql/common/var/lib/postgresql
OSError: [Errno 16] Device or resource busy
```

## Expected behavior

All 3 units should become active and form a healthy Patroni cluster with 1 leader and 2 sync standbys within a few minutes.

## Actual behavior

The replica units get permanently stuck in a bootstrap loop and never join the cluster. The primary shows "Primary (degraded)" and the replicas show "awaiting for member to start" or "waiting to start PostgreSQL".

### Root cause analysis

There are three issues that combine to create a permanent deadlock:

**1. `pg_basebackup` fails on the replica**: When new units are added, Patroni starts on the replica and immediately attempts `pg_basebackup` to bootstrap from the leader. On MicroCloud (doubly-nested LXD containers), the leader's PostgreSQL rejects connections with a shared memory permission error:

```
2026-03-05 18:30:01 UTC [15489]: user=replication,db=[unknown],app=[unknown],client=240.122.0.116,line=3
FATAL: 42501: could not open shared memory segment "/PostgreSQL.992557020": Permission denied
```

This causes `pg_basebackup` to exit with code 1. On standard LXD with non-default storage pools, the failure can also be caused by a `pg_hba.conf` race condition where the leader's replication rules have not yet been updated to include the new replica's IP (the charm's `peers_ips` is empty when the first replica connects).

**2. Mount point prevents cleanup (EBUSY)**: After `pg_basebackup` fails, Patroni attempts to clean up the data directory by calling `shutil.rmtree()` and `os.rename()` on the PostgreSQL data directory (`/var/snap/charmed-postgresql/common/var/lib/postgresql`). However, this directory is a Juju storage mount point, so both operations fail with `OSError: [Errno 16] Device or resource busy`. This means:

- The data directory cannot be deleted or renamed
- On MicroCloud, `pg_basebackup` continues to fail on every retry due to the persistent shared memory error on the leader
- On standard LXD, leftover files/directories from the failed `pg_basebackup` may also prevent subsequent attempts from succeeding, turning a transient `pg_hba.conf` race into a permanent failure
- This creates a permanent loop

**3. Cascading deadlock**: The leader's `_add_members()` method calls `add_cluster_member()` which checks `are_all_members_ready()`. Since the stuck unit is perpetually in "creating replica" state, this check fails, preventing the leader from reconfiguring the cluster. Any additional units added after this point will also be stuck because they never get their Patroni started.

## Versions

Operating system: Ubuntu 24.04 LTS

Juju CLI: 3.6.14

Juju agent: 3.6.14

Charm revision: 1047 (16/stable)

LXD: 5.21.4

## Log output

Patroni log from the stuck replica unit (repeating indefinitely):

```
2026-03-05 12:19:59 UTC [11408]: INFO: Lock owner: postgresql-0; I am postgresql-1
2026-03-05 12:19:59 UTC [11408]: INFO: trying to bootstrap from leader 'postgresql-0'
2026-03-05 12:19:59 UTC [11408]: INFO: bootstrap from leader 'postgresql-0' in progress
2026-03-05 12:19:59 UTC [11408]: ERROR: Error when fetching backup: pg_basebackup exited with code=1
2026-03-05 12:19:59 UTC [11408]: WARNING: Trying again in 5 seconds
2026-03-05 12:20:04 UTC [11408]: ERROR: Error when fetching backup: pg_basebackup exited with code=1
2026-03-05 12:20:04 UTC [11408]: ERROR: failed to bootstrap from leader 'postgresql-0'
2026-03-05 12:20:04 UTC [11408]: INFO: Removing data directory: /var/snap/charmed-postgresql/common/var/lib/postgresql
2026-03-05 12:20:04 UTC [11408]: ERROR: Could not remove data directory /var/snap/charmed-postgresql/common/var/lib/postgresql
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/patroni/postgresql/__init__.py", line 1346, in remove_data_directory
shutil.rmtree(self._data_dir)
File "/usr/lib/python3.12/shutil.py", line 796, in rmtree
onexc(os.rmdir, path, err)
File "/usr/lib/python3.12/shutil.py", line 794, in rmtree
os.rmdir(path, dir_fd=dir_fd)
OSError: [Errno 16] Device or resource busy: '/var/snap/charmed-postgresql/common/var/lib/postgresql'
2026-03-05 12:20:04 UTC [11408]: INFO: renaming data directory to /var/snap/charmed-postgresql/common/var/lib/postgresql.failed
2026-03-05 12:20:04 UTC [11408]: ERROR: Could not rename data directory /var/snap/charmed-postgresql/common/var/lib/postgresql
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/patroni/postgresql/__init__.py", line 1346, in remove_data_directory
shutil.rmtree(self._data_dir)
File "/usr/lib/python3.12/shutil.py", line 796, in rmtree
onexc(os.rmdir, path, err)
File "/usr/lib/python3.12/shutil.py", line 794, in rmtree
os.rmdir(path, dir_fd=dir_fd)
OSError: [Errno 16] Device or resource busy: '/var/snap/charmed-postgresql/common/var/lib/postgresql'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/patroni/postgresql/__init__.py", line 1319, in move_data_directory
os.rename(self._data_dir, new_name)
OSError: [Errno 16] Device or resource busy: '/var/snap/charmed-postgresql/common/var/lib/postgresql' -> '/var/snap/charmed-postgresql/common/var/lib/postgresql.failed'
```

## Additional context

- **100% reproduction rate on MicroCloud.** Reproduced 8 out of 8 attempts on a 3-node MicroCloud cluster with `dir` storage. The bug also occurs on standard LXD with non-default storage pools (btrfs, zfs) but is intermittent there (~60% rate).
- Originally reported by a user running MicroCloud (LXD 5.21.4) with default LXD local storage.
- On MicroCloud, even with `dir` storage, Juju creates bind-mounted storage volumes for the PostgreSQL data directory, triggering the EBUSY condition when Patroni tries to clean up after a failed `pg_basebackup`.
- On MicroCloud, the primary trigger for the `pg_basebackup` failure is a shared memory permission error on the leader (`could not open shared memory segment: Permission denied`), caused by the doubly-nested container environment. On standard LXD, the trigger is typically a `pg_hba.conf` race condition.
- With the default `dir` storage pool on a standard LXD controller, the race condition may still occur but Patroni can sometimes recover because the data directory is not a mount point and `shutil.rmtree()` succeeds.
- The fix for DPE-9150 (present in rev 1047) does not address this issue.