canonical / canonical/postgresql-operator
`database-peers` relation missing `tls=enabled` on one unit causes HTTP health checks against HTTPS Patroni API
- Dominant language
- Python
- Stars
- 20
- Forks
- 36
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 30
Description
In a 3-unit postgresql cluster, integrated with self-signed-certificates for TLS, two units had `tls: enabled` in the database-peers relation data while the third unit did not. This manifested as "member awaiting to start" on the problematic unit and the following errors on patroni logs of each unit, which showed something was sending plain HTTP to the Patroni API:
```
...
SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca
...
SSL: HTTP_REQUEST] http request
```
On the affected unit, the Patroni API had TLS configured and was serving the expected certificate (tested manually), but the charm was unaware of the TLS configuration and was attempting Patroni health checks over http on port 8008, leading on the repeated errors mentioned above.
Following the recommendations of @marceloneppel and @dragomirp, the following workaround solved the issue:
```
# check for units missing "tls: enabled"
juju show-unit postgresql/X --endpoint database-peers
# To grep the relation id for the next command.
juju show-unit postgresql/X --endpoint=database-peers | grep relation-id
juju exec --unit postgresql/X 'relation-set -r RELATION-ID-FROM-ABOVE tls="enabled"'
```
where postgresql/X was the faulty unit. After this, the charm started using https for Patroni, Patroni stopped logging errors, health checks began working correctly and the unit appeared healthy in `juju status`
## Expected behavior
When TLS is enabled for the application (via the certificates relation/self-signed-certificates) and Patroni is configured with TLS, all units should have consistent `tls: enabled` state in the `database-peers` relation.
## Actual behavior
Occasionally, units end up with Patroni configured with TLS but no `tls: enabled` flag on the `database-peers` relation.
## Versions
Juju CLI: 3.6.11
Juju agent: 3.6.11
Charm revision: 14/stable rev 553, base 22.04
self-signed-certificates: latest/edge, rev 419
## Log output
Juju debug log:
tcpdump on the affected unit
```
GET /cluster HTTP/1.1
Host: :8008
User-Agent: python-requests/2.32.3
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Basic
GET /health HTTP/1.1
Host: 10.146.65.77:8008
User-Agent: python-requests/2.32.3
...
Authorization: Basic
```
`ss -tnp "dport = :8008"` while seeing the error
```
SYN-SENT 0 1 10.146.65.77:46374 10.146.65.69:8008 users:(("python",pid=1058948,fd=4))
SYN-SENT 0 1 10.146.65.77:46080 10.146.65.69:8008 users:(("python",pid=1058948,fd=4))
CLOSE-WAIT 361 0 10.146.65.77:49812 10.146.65.72:8008 users:(("python",pid=1058948,fd=4))
$ ps aux | grep 1058948
root 1058948 0.5 0.9 167436 74708 ? Sl 14:52 0:00 /var/lib/juju/agents/unit-postgresql-2/charm/venv/bin/python /var/lib/juju/agents/unit-postgresql-2/charm/src/charm.py
```
[juju debug log replay.txt](https://github.com/user-attachments/files/23682967/juju.debug.log.replay.txt)
## Additional context
Contributor guide
Assessment
This issue has not been assessed yet.