world-postgres: stream readers can stall after LISTEN disconnects or missed NOTIFY event
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Bug
@workflow/world-postgres currently relies on PostgreSQL LISTEN/NOTIFY for live stream chunk delivery.
This is fragile: NOTIFY is only a wake-up signal for currently connected listeners, not a durable backlog. If the dedicated LISTEN workflow_event_chunk client disconnects, or if a notification is missed during reconnect, chunks can still be written to the streams table successfully while active readers stop receiving live updates indefinitely.
In other words: the streams table is the source of truth, and LISTEN/NOTIFY should only be used to wake readers up to re-query chunks newer than their last delivered chunk_id.
Symptoms
In production, after the dedicated LISTEN client is dropped:
writeToStream(...)continues inserting chunk rows successfully.pg_notify(...)continues executing successfully.readFromStream(...)readers may receive the initial query batch, then never receive subsequent chunks.- Restarting the pod restores delivery until the next LISTEN disconnect.
This silently halts live in-process delivery in the affected process, while persisted stream rows remain intact.
Proposed fix
This needs two layers:
-
Make
listenChannelresilient:- attach
errorandendhandlers to the dedicatedpg.Client - reconnect with bounded exponential backoff
- re-run
LISTEN workflow_event_chunkafter reconnect - stop reconnect attempts on
close()
- attach
-
Make
readFromStreamresilient to missed notifications:- keep a per-reader
lastChunkId - load initial chunks from the
streamstable - on notification, query
streams WHERE chunk_id > lastChunkId - periodically run the same query as a polling fallback
- dedupe/order by
chunk_id - stop polling on EOF, cancel, or controller close
- keep a per-reader
This makes world-postgres stream delivery durable even when the LISTEN connection is interrupted.
Relation to other work
This is compatible with #1847, but it is a lower-level world-postgres reliability issue. Core-level stream reconnect cannot recover notifications that PostgreSQL never delivered to a disconnected LISTEN client. The Postgres world still needs to treat the table as the durable source of truth.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with listenChannel, readFromStream, and writeToStream in @workflow/world-postgres, treating the streams table as the durable source of truth. Trace how notifications and chunk IDs flow through readers, then verify that reconnects, missed notifications, polling, ordering, deduplication, EOF, cancellation, and close all stop or recover delivery as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100