canonical / canonical/postgresql-operator

Patroni and health checks proxy handling

Open
#1,714 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

Despite Patroni being healthy:
```
root@juju-78f33d-0:/var/lib/juju/agents/unit-postgresql-0/charm# curl -sk https://10.32.69.4:8008/cluster | jq .
{
"members": [
{
"name": "postgresql-0",
"role": "leader",
"state": "running",
"api_url": "https://10.32.69.4:8008/patroni",
"host": "10.32.69.4",
"port": 5432,
"timeline": 1
}
],
"scope": "postgresql"
}

root@juju-78f33d-0:/var/lib/juju/agents/unit-postgresql-0/charm# curl -sk https://10.32.69.4:8008/switchover | jq .
{
"state": "running",
"postmaster_start_time": "2026-05-26 10:11:01.122042+00:00",
"role": "master",
"server_version": 160013,
"xlog": {
"location": 24840288
},
"timeline": 1,
"dcs_last_seen": 1779791222,
"database_system_identifier": "7644140955326675877",
"patroni": {
"version": "3.3.8",
"scope": "postgresql",
"name": "postgresql-0"
}
}
```

It seems `httpx.AsyncClient` (v0.28+) respects HTTPS_PROXY environment variables by default and routes the Patroni API request through the HTTP proxy, while `requests` (used for the health check) also respect them but handle no_proxy CIDR notation differently. Resulting in the leader unit `awaiting start of the primary`:

```
May 26 10:10:55 juju-78f33d-0 systemd[1]: Started snap.charmed-postgresql.patroni.service - Service for snap application charmed-postgresql.patroni.
░░ Subject: A start job for unit snap.charmed-postgresql.patroni.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit snap.charmed-postgresql.patroni.service has finished successfully.
░░
░░ The job identifier is 3046.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The files belonging to this database system will be owned by user "_daemon_".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: This user must also own the server process.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The database cluster will be initialized with locale "en_US.UTF-8".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The default text search configuration will be set to "english".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: Data page checksums are enabled.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: fixing permissions on existing directory /var/snap/charmed-postgresql/common/var/lib/postgresql ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: fixing permissions on existing directory /var/snap/charmed-postgresql/common/data/logs ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: creating subdirectories ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting dynamic shared memory implementation ... posix
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default max_connections ... 100
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default shared_buffers ... 128MB
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default time zone ... Etc/UTC
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: creating configuration files ... ok
May 26 10:10:59 juju-78f33d-0 charmed-postgresql.patroni[7069]: running bootstrap script ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: performing post-bootstrap initialization ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: syncing data to disk ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: initdb: warning: enabling "trust" authentication for local connections
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: Success. You can now start the database server using:
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: /snap/charmed-postgresql/current/usr/lib/postgresql/16/bin/pg_ctl -D /var/snap/charmed-postgresql/common/var/lib/postgresql -l logfile start
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7087]: 10.32.69.4:5432 - no response
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:00 UTC [7086]: user=,db=,app=,client=,line=1 LOG: 00000: pgaudit extension initialized
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:00 UTC [7086]: user=,db=,app=,client=,line=2 LOCATION: _PG_init, pgaudit.c:2263
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=3 LOG: 00000: redirecting log output to logging collector process
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=4 HINT: Future log output will appear in directory "/var/snap/charmed-postgresql/common/var/log/postgresql".
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=5 LOCATION: SysLogger_Start, syslogger.c:712
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7098]: 10.32.69.4:5432 - accepting connections
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7100]: 10.32.69.4:5432 - accepting connections
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: Exception in thread Thread-43 (process_request_thread):
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: Traceback (most recent call last):
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: self.run()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: File "/usr/lib/python3.12/threading.py", line 1010, in run
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: self._target(*self._args, **self._kwargs)
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: File "/usr/lib/python3/dist-packages/patroni/api.py", line 1685, in process_request_thread
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: request.do_handshake()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: File "/usr/lib/python3.12/ssl.py", line 1320, in do_handshake
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: self._sslobj.do_handshake()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: ssl.SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:1000)
```

Steps to reproduce:
```
$ juju add model database
$ juju model-config http-proxy=http://squid.internal:3128
$ juju model-config https-proxy=http://squid.internal:3128
$ juju model-config no-proxy="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"

$ juju model-config cloudinit-userdata='''#cloud-config
write_files:
- path: /etc/environment
permissions: '0644'
owner: root:root
content: |
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
http_proxy=http://10.22.2.1:3128
https_proxy=http://10.22.2.1:3128
no_proxy="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"
HTTP_PROXY=http://10.22.2.1:3128
HTTPS_PROXY=http://10.22.2.1:3128
NO_PROXY="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"
'''
$ juju deploy postgresql -n 3

$ juju status
App Version Status Scale Charm Channel Rev Exposed Message
postgresql 16.13 active 3 postgresql 16/stable 1089 no

Unit Workload Agent Machine Public address Ports Message
postgresql/0* waiting idle 0 10.32.69.4 5432/tcp awaiting start of the primary
postgresql/1 waiting idle 2 10.32.69.6 5432/tcp awaiting for cluster to start
postgresql/2 waiting idle 4 10.32.69.10 5432/tcp awaiting for cluster to start

Machine State Address Inst id Base AZ Message
0 started 10.32.69.4 juju-78f33d-0 ubuntu@24.04 pc8b-infra Running
2 started 10.32.69.6 juju-78f33d-2 ubuntu@24.04 pc8b-infra Running
4 started 10.32.69.10 juju-78f33d-4 ubuntu@24.04 pc8b-infra Running

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.