vercel / vercel/workflow

world-postgres: a producer-only World — enqueue without a runner or startup recovery

Open
#3,968 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Summary

@workflow/world-postgres has no way to build a World that only enqueues. Every process that calls start() on a run — even one that will never execute a step — gets a graphile-worker runner and runs reenqueueActiveRuns(), because queue() awaits start() and start() does both. A producer-only mode (no runner, no startup recovery) would let the common "API process starts runs, worker process executes them" topology work without side effects.

Setup

Two deployment units share one Postgres World:

  • an API process that calls start(workflow, args, { world }) when a user submits work, N replicas, rolled frequently;
  • a worker process that compiled the "use workflow" / "use step" code and serves .well-known/workflow/v1/*. It is the only unit that can execute anything. WORKFLOW_LOCAL_BASE_URL in the API points at the worker's Kubernetes Service.

Versions: @workflow/world-postgres@4.3.5, @workflow/world@4.5.0, workflow@4.8.4, graphile-worker@0.16.6, Node 24. The same coupling is on main at 22a9668dcf51f9799c26be0a4c144670df46101e:

  • packages/world-postgres/src/queue.ts:497queue() begins with await start();
  • packages/world-postgres/src/index.ts:86-88start() is queue.start() then reenqueueActiveRuns(...);
  • packages/world-postgres/src/queue.ts:632start() ends in graphile's run({ taskList });
  • packages/world-postgres/src/config.tsPostgresWorldConfig offers jobPrefix, namespace, queueConcurrency, applicationManagedShutdown, streamFlushIntervalMs; nothing selects a role.

What happens

Because the API only ever wants to enqueue, everything its embedded runner does is unwanted:

  1. It executes jobs by POSTing across pods. During a rolling update of the worker with no ready endpoint, the API's runner claims the due jobs, its POSTs fail, and with maxAttempts: 3 (4.3.5 queue.ts:115) the job is permanently failed within a few seconds. The worker's own runner has no such window — it dies with its executor and the job simply waits. The failed job is revived by the next start() anywhere (recovery re-enqueues it), so the run is stuck for roughly the rollout, not forever — but the failure is manufactured entirely by a consumer that should not exist.
  2. Every API replica runs startup recovery. N replicas rolling = N full re-enqueues of every active run, on top of the worker's own. #3119 covers the duplicate accumulation and #3758 the concurrent-replay corruption this can cause against a live executor; a producer-only API removes N of the N+1 sources of both without waiting for the recovery predicate to be fixed.
  3. Each API replica holds a pool and a LISTEN/poll loop against the World database, and graphile's log lines land in the API's log.

Proposal

A config option on PostgresWorldConfig — e.g. role: 'producer' | 'worker' (default 'worker', current behaviour) — such that under 'producer':

  • start() still runs workerUtils.migrate() (or skips it, if you prefer that a producer never migrates) but never calls graphile's run() and never calls reenqueueActiveRuns();
  • queue() works as today;
  • close() only ends what was opened.

The runner-side startRunnerWhenExecutorIsReady loopback check already acknowledges that the enqueueing process and the executing process can differ; this option would make that split explicit.

Workaround today

pnpm patch on world-postgres guarding run() and reenqueueActiveRuns() behind a config flag. Filing so the split can be first-class rather than a local patch re-applied on every upgrade.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/world-postgres/src/queue.ts, especially queue() and the runner startup path, then read packages/world-postgres/src/index.ts and config.ts. Trace how start(), reenqueueActiveRuns(), graphile's run(), queue(), and close() currently interact. Done means a configured producer can enqueue without starting a runner or recovery while preserving current worker behavior and correctly closing resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.