canonical / canonical/postgresql-operator

Unexpected exception in Patroni: BlockingIOError: [Errno 11] Resource temporarily unavailable

Open
#1,859 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
20
Forks
36
Avg merge
4d 15h
Merged PRs (30d)
30

Description

## Steps to reproduce
Not really sure to be honest... This seems to have happened out of the blue in a deployment in PS7. Reporting it here because the charm logged `Unexpected exception raised, please report it as a BUG`.

I reported this separately from https://github.com/canonical/postgresql-operator/issues/1358 because no units were removed, AFAIK.

## Expected behavior
Units auto-recover from whatever caused them to break.

## Actual behavior
Cluster is unstable. It is impossible to establish CMRs with it.

## Versions
Sorry, we would need IS for this. This DBaaS cluster was provisioned in late June, if it helps.

## Log output
Full logs with output from units 0 and 2 in the cluster (unit 1 seems to be missing for some reason): https://pastebin.canonical.com/p/XwPpqnrZNb/

Relevant segment:
```
[postgresql postgresql/2] 2026-07-14 08:34:29 UTC [3749374]: user=[unknown],db=[unknown],app=[unknown],client=10.156.72.47,line=1 LOG: 00000: connection received: host=10.156.72.47 port=55322
[postgresql postgresql/2] 2026-07-14 08:34:29 UTC [3749374]: user=[unknown],db=[unknown],app=[unknown],client=10.156.72.47,line=2 LOCATION: BackendInitialize, postmaster.c:4354
[postgresql postgresql/2] 2026-07-14 08:34:29 UTC [3749374]: user=operator,db=postgres,app=[unknown],client=10.156.72.47,line=3 FATAL: 57P03: the database system is starting up
[postgresql postgresql/2] 2026-07-14 08:34:29 UTC [3749374]: user=operator,db=postgres,app=[unknown],client=10.156.72.47,line=4 LOCATION: ProcessStartupPacket, postmaster.c:2336
[postgresql postgresql/2] 2026-07-14 08:34:31 UTC [3749384]: user=,db=,app=,client=,line=1 FATAL: 08006: could not connect to the primary server: connection to server at "10.156.72.53", port 5432 failed: Connection refused
[postgresql postgresql/2] Is the server running on that host and accepting TCP/IP connections?
[postgresql postgresql/2] 2026-07-14 08:34:31 UTC [3749384]: user=,db=,app=,client=,line=2 LOCATION: WalReceiverMain, walreceiver.c:303
[postgresql postgresql/2] 2026-07-14 08:34:31 UTC [2802160]: user=,db=,app=,client=,line=726719 LOG: 00000: new timeline 20 forked off current database system timeline 18 before current recovery point 0/4A000028
[postgresql postgresql/2] 2026-07-14 08:34:31 UTC [2802160]: user=,db=,app=,client=,line=726720 LOCATION: rescanLatestTimeLine, xlogrecovery.c:4108
[postgresql postgresql/2] 2026-07-14 08:34:31 UTC [2802160]: user=,db=,app=,client=,line=726721 LOG: 00000: waiting for WAL to become available at 0/4A000040
[postgresql postgresql/2] 2026-07-14 08:34:31 UTC [2802160]: user=,db=,app=,client=,line=726722 LOCATION: WaitForWALToBecomeAvailable, xlogrecovery.c:3649
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: INFO: Unexpected exception raised, please report it as a BUG
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: ERROR: Unexpected exception
[postgresql postgresql/2] Traceback (most recent call last):
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/patroni/ha.py", line 2108, in run_cycle
[postgresql postgresql/2] info = self._run_cycle()
[postgresql postgresql/2] ^^^^^^^^^^^^^^^^^
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/patroni/ha.py", line 2062, in _run_cycle
[postgresql postgresql/2] self.touch_member()
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/patroni/ha.py", line 482, in touch_member
[postgresql postgresql/2] ret = self.dcs.touch_member(data)
[postgresql postgresql/2] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/patroni/dcs/raft.py", line 439, in touch_member
[postgresql postgresql/2] return self._sync_obj.set(self.member_path, value, self._ttl, timeout=2) is not False
[postgresql postgresql/2] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/patroni/dcs/raft.py", line 204, in set
[postgresql postgresql/2] return self.retry(self._set, key, data, **kwargs)
[postgresql postgresql/2] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/patroni/dcs/raft.py", line 165, in retry
[postgresql postgresql/2] func(*args, **kwargs)
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/pysyncobj/syncobj.py", line 1481, in newFunc
[postgresql postgresql/2] applier(pickle.dumps(cmd), callback, _COMMAND_TYPE.REGULAR)
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/pysyncobj/syncobj.py", line 427, in _applyCommand
[postgresql postgresql/2] self.__pipeNotifier.notify()
[postgresql postgresql/2] File "/usr/lib/python3/dist-packages/pysyncobj/pipe_notifier.py", line 22, in notify
[postgresql postgresql/2] os.write(self.__pipeW, b'o')
[postgresql postgresql/2] BlockingIOError: [Errno 11] Resource temporarily unavailable
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: INFO: establishing a new patroni heartbeat connection to postgres
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: WARNING: Retry got exception: connection problems
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: WARNING: Failed to determine PostgreSQL state from the connection, falling back to cached role
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: INFO: Lock owner: postgresql-0; I am postgresql-2
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: INFO: Still starting up as a standby.
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: INFO: Lock owner: postgresql-0; I am postgresql-2
[postgresql postgresql/2] 2026-07-14 08:34:32 UTC [2801490]: INFO: establishing a new patroni heartbeat connection to postgres
[postgresql postgresql/2] 2026-07-14 08:34:36 UTC [3749430]: user=,db=,app=,client=,line=1 FATAL: 08006: could not connect to the primary server: connection to server at "10.156.72.53", port 5432 failed: Connection refused
[postgresql postgresql/2] Is the server running on that host and accepting TCP/IP connections?
[postgresql postgresql/2] 2026-07-14 08:34:36 UTC [3749430]: user=,db=,app=,client=,line=2 LOCATION: WalReceiverMain, walreceiver.c:303
[postgresql postgresql/2] 2026-07-14 08:34:36 UTC [2802160]: user=,db=,app=,client=,line=726723 LOG: 00000: new timeline 20 forked off current database system timeline 18 before current recovery point 0/4A000028
[postgresql postgresql/2] 2026-07-14 08:34:36 UTC [2802160]: user=,db=,app=,client=,line=726724 LOCATION: rescanLatestTimeLine, xlogrecovery.c:4108
[postgresql postgresql/2] 2026-07-14 08:34:36 UTC [2802160]: user=,db=,app=,client=,line=726725 LOG: 00000: waiting for WAL to become available at 0/4A000040
[postgresql postgresql/2] 2026-07-14 08:34:36 UTC [2802160]: user=,db=,app=,client=,line=726726 LOCATION: WaitForWALToBecomeAvailable, xlogrecovery.c:3649
```

## Additional context
This has been reported internally here: https://warthogs.atlassian.net/browse/ISREQ-3825

Contributor guide

Open the contributing guide

Research direction

Start with the traceback in patroni/ha.py, following touch_member into patroni/dcs/raft.py and the pysyncobj pipe_notifier.py call shown in the logs. Review the linked full logs and deployment context, then determine why the BlockingIOError occurs and whether the cluster can recover without manual intervention. Done means the failure is reproducible or explained and recovery behavior is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.