world-postgres: a producer-only World — enqueue without a runner or startup recovery
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Summary
@workflow/world-postgres has no way to build a World that only enqueues. Every process that calls start() on a run — even one that will never execute a step — gets a graphile-worker runner and runs reenqueueActiveRuns(), because queue() awaits start() and start() does both. A producer-only mode (no runner, no startup recovery) would let the common "API process starts runs, worker process executes them" topology work without side effects.
Setup
Two deployment units share one Postgres World:
- an API process that calls
start(workflow, args, { world })when a user submits work, N replicas, rolled frequently; - a worker process that compiled the
"use workflow"/"use step"code and serves.well-known/workflow/v1/*. It is the only unit that can execute anything.WORKFLOW_LOCAL_BASE_URLin the API points at the worker's Kubernetes Service.
Versions: @workflow/world-postgres@4.3.5, @workflow/world@4.5.0, workflow@4.8.4, graphile-worker@0.16.6, Node 24. The same coupling is on main at 22a9668dcf51f9799c26be0a4c144670df46101e:
packages/world-postgres/src/queue.ts:497—queue()begins withawait start();packages/world-postgres/src/index.ts:86-88—start()isqueue.start()thenreenqueueActiveRuns(...);packages/world-postgres/src/queue.ts:632—start()ends in graphile'srun({ taskList });packages/world-postgres/src/config.ts—PostgresWorldConfigoffersjobPrefix,namespace,queueConcurrency,applicationManagedShutdown,streamFlushIntervalMs; nothing selects a role.
What happens
Because the API only ever wants to enqueue, everything its embedded runner does is unwanted:
- It executes jobs by POSTing across pods. During a rolling update of the worker with no ready endpoint, the API's runner claims the due jobs, its POSTs fail, and with
maxAttempts: 3(4.3.5queue.ts:115) the job is permanently failed within a few seconds. The worker's own runner has no such window — it dies with its executor and the job simply waits. The failed job is revived by the nextstart()anywhere (recovery re-enqueues it), so the run is stuck for roughly the rollout, not forever — but the failure is manufactured entirely by a consumer that should not exist. - Every API replica runs startup recovery. N replicas rolling = N full re-enqueues of every active run, on top of the worker's own. #3119 covers the duplicate accumulation and #3758 the concurrent-replay corruption this can cause against a live executor; a producer-only API removes N of the N+1 sources of both without waiting for the recovery predicate to be fixed.
- Each API replica holds a pool and a
LISTEN/poll loop against the World database, and graphile's log lines land in the API's log.
Proposal
A config option on PostgresWorldConfig — e.g. role: 'producer' | 'worker' (default 'worker', current behaviour) — such that under 'producer':
start()still runsworkerUtils.migrate()(or skips it, if you prefer that a producer never migrates) but never calls graphile'srun()and never callsreenqueueActiveRuns();queue()works as today;close()only ends what was opened.
The runner-side startRunnerWhenExecutorIsReady loopback check already acknowledges that the enqueueing process and the executing process can differ; this option would make that split explicit.
Workaround today
pnpm patch on world-postgres guarding run() and reenqueueActiveRuns() behind a config flag. Filing so the split can be first-class rather than a local patch re-applied on every upgrade.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/world-postgres/src/queue.ts, especially queue() and the runner startup path, then read packages/world-postgres/src/index.ts and config.ts. Trace how start(), reenqueueActiveRuns(), graphile's run(), queue(), and close() currently interact. Done means a configured producer can enqueue without starting a runner or recovery while preserving current worker behavior and correctly closing resources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, typescript
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100