vercel / vercel/workflow

Shared Postgres world: queue namespaces don't isolate claiming, and startup recovery re-enqueues other applications' active runs

Open
#2,978 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Summary

When multiple independent applications share one @workflow/world-postgres database, they execute each other's workflow runs. Queue namespaces (WORKFLOW_QUEUE_NAMESPACE) do not prevent this, for two independent reasons:

  1. Claiming ignores namespaces. The graphile-worker task identifiers are ${jobPrefix}flows / ${jobPrefix}steps — the namespace only prefixes the topic string inside the message. Any consumer polling the same database claims any namespace's jobs, and createTaskHandler then reconstructs the queue name using the consumer's own prefix (const queueName = ${queue}${messageData.id}``), so the foreign job is executed locally instead of being skipped.
  2. Startup recovery is unscoped. reenqueueActiveRuns lists all pending/running runs with no namespace/deployment filter and re-enqueues each under the booting application's namespace prefix. So every application boot takes over every other application's in-flight runs. #2888 fixed which namespace the recovered runs are enqueued under, and #2667 scoped world-local recovery by tag — but the Postgres world still passes its raw, unfiltered storage.runs into recovery (packages/world-postgres's start()), so cross-application takeover remains. Notably, workflow_runs.deployment_id exists in the schema but world-postgres hardcodes getDeploymentId() to 'postgres', so it can't be used for scoping today.

Code references (checked against @workflow/world@5.0.0-beta.21 and @workflow/world-postgres@5.0.0-beta.27, the latest betas as of 2026-07-17):

  • packages/world/src/recovery.tsruns.list({ status, resolveData: 'none', ... }) with no scoping parameter.
  • packages/world-postgres/src/queue.tsgetJobQueueName() derives the task identifier from jobPrefix only; createTaskHandler() rebuilds the queue name from the local prefix without validating the producer's namespace.
  • packages/world-postgres/src/index.tsstart()reenqueueActiveRuns(storage.runs, queue.queue, 'world-postgres', config.namespace).

How we hit this

We run a self-hosted platform that hosts many independent Eve agents (each is its own app/process), and initially pointed all of them at one WORKFLOW_POSTGRES_URL. Eve already derives a distinct WORKFLOW_QUEUE_NAMESPACE per agent, so topics were namespaced — yet agents intermittently executed each other's turns with the wrong application's code and credentials (surfacing as flaky provider/API-key errors, depending on which runner won the claim). Tracing it led to the two mechanisms above.

Repro sketch

  1. Two apps A and B (any two workflow apps with same-named workflows — e.g. two instances of the same framework), same WORKFLOW_POSTGRES_URL, distinct WORKFLOW_QUEUE_NAMESPACE.
  2. Start a long-running workflow in A.
  3. While it is running, boot B. B logs [world-postgres] Re-enqueued N active run(s) on startup including A's run, under B's namespace.
  4. B's runner claims and executes A's run (same workflow_flows task id; handler reconstructs the topic with B's prefix). With distinct WORKFLOW_POSTGRES_JOB_PREFIX values the takeover becomes deterministic: the re-enqueued foreign run can only be claimed by B.

Suggested directions

Either would resolve this class of issue:

  • Scope claiming and recovery to the world's identity — e.g. include the namespace in the graphile task identifier (or validate the producer namespace in the handler and re-enqueue/skip on mismatch), and give runs.list a filter that reenqueueActiveRuns passes through (the deployment_id column looks like a natural hook, similar in spirit to the world-local tag scoping in #2667).
  • Or, if a Postgres world database is intended to be exclusive to a single application, state that explicitly in the world-postgres docs — today nothing fails loudly when two applications share one database; they just silently cross-execute.

Workaround

We moved to one Postgres database per application (derived from a base URL, created + workflow-postgres-setup bootstrapped on demand). That fully isolates storage, queue, and recovery, and has been working well — sharing it here for other self-hosters who hit the same symptom.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/world-postgres/src/queue.ts and src/index.ts, then trace recovery in packages/world/src/recovery.ts. Reproduce the two-application scenario with one Postgres database and distinct queue namespaces, comparing claiming with startup recovery. Done means applications sharing a database cannot claim or re-enqueue another application's active runs, or the documentation clearly states that the database must be exclusive.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, typescript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.