vercel / vercel/workflow

world-postgres: stream readers can stall after LISTEN disconnects or missed NOTIFY event

Open
#1,855 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Bug

@workflow/world-postgres currently relies on PostgreSQL LISTEN/NOTIFY for live stream chunk delivery.

This is fragile: NOTIFY is only a wake-up signal for currently connected listeners, not a durable backlog. If the dedicated LISTEN workflow_event_chunk client disconnects, or if a notification is missed during reconnect, chunks can still be written to the streams table successfully while active readers stop receiving live updates indefinitely.

In other words: the streams table is the source of truth, and LISTEN/NOTIFY should only be used to wake readers up to re-query chunks newer than their last delivered chunk_id.

Symptoms

In production, after the dedicated LISTEN client is dropped:

  • writeToStream(...) continues inserting chunk rows successfully.
  • pg_notify(...) continues executing successfully.
  • readFromStream(...) readers may receive the initial query batch, then never receive subsequent chunks.
  • Restarting the pod restores delivery until the next LISTEN disconnect.

This silently halts live in-process delivery in the affected process, while persisted stream rows remain intact.

Proposed fix

This needs two layers:

  1. Make listenChannel resilient:

    • attach error and end handlers to the dedicated pg.Client
    • reconnect with bounded exponential backoff
    • re-run LISTEN workflow_event_chunk after reconnect
    • stop reconnect attempts on close()
  2. Make readFromStream resilient to missed notifications:

    • keep a per-reader lastChunkId
    • load initial chunks from the streams table
    • on notification, query streams WHERE chunk_id > lastChunkId
    • periodically run the same query as a polling fallback
    • dedupe/order by chunk_id
    • stop polling on EOF, cancel, or controller close

This makes world-postgres stream delivery durable even when the LISTEN connection is interrupted.

Relation to other work

This is compatible with #1847, but it is a lower-level world-postgres reliability issue. Core-level stream reconnect cannot recover notifications that PostgreSQL never delivered to a disconnected LISTEN client. The Postgres world still needs to treat the table as the durable source of truth.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with listenChannel, readFromStream, and writeToStream in @workflow/world-postgres, treating the streams table as the durable source of truth. Trace how notifications and chunk IDs flow through readers, then verify that reconnects, missed notifications, polling, ordering, deduplication, EOF, cancellation, and close all stop or recover delivery as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.