Shared Postgres world: queue namespaces don't isolate claiming, and startup recovery re-enqueues other applications' active runs
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Summary
When multiple independent applications share one @workflow/world-postgres database, they execute each other's workflow runs. Queue namespaces (WORKFLOW_QUEUE_NAMESPACE) do not prevent this, for two independent reasons:
- Claiming ignores namespaces. The graphile-worker task identifiers are
${jobPrefix}flows/${jobPrefix}steps— the namespace only prefixes the topic string inside the message. Any consumer polling the same database claims any namespace's jobs, andcreateTaskHandlerthen reconstructs the queue name using the consumer's own prefix (const queueName =${queue}${messageData.id}``), so the foreign job is executed locally instead of being skipped. - Startup recovery is unscoped.
reenqueueActiveRunslists all pending/running runs with no namespace/deployment filter and re-enqueues each under the booting application's namespace prefix. So every application boot takes over every other application's in-flight runs. #2888 fixed which namespace the recovered runs are enqueued under, and #2667 scoped world-local recovery by tag — but the Postgres world still passes its raw, unfilteredstorage.runsinto recovery (packages/world-postgres'sstart()), so cross-application takeover remains. Notably,workflow_runs.deployment_idexists in the schema but world-postgres hardcodesgetDeploymentId()to'postgres', so it can't be used for scoping today.
Code references (checked against @workflow/world@5.0.0-beta.21 and @workflow/world-postgres@5.0.0-beta.27, the latest betas as of 2026-07-17):
packages/world/src/recovery.ts—runs.list({ status, resolveData: 'none', ... })with no scoping parameter.packages/world-postgres/src/queue.ts—getJobQueueName()derives the task identifier fromjobPrefixonly;createTaskHandler()rebuilds the queue name from the local prefix without validating the producer's namespace.packages/world-postgres/src/index.ts—start()→reenqueueActiveRuns(storage.runs, queue.queue, 'world-postgres', config.namespace).
How we hit this
We run a self-hosted platform that hosts many independent Eve agents (each is its own app/process), and initially pointed all of them at one WORKFLOW_POSTGRES_URL. Eve already derives a distinct WORKFLOW_QUEUE_NAMESPACE per agent, so topics were namespaced — yet agents intermittently executed each other's turns with the wrong application's code and credentials (surfacing as flaky provider/API-key errors, depending on which runner won the claim). Tracing it led to the two mechanisms above.
Repro sketch
- Two apps A and B (any two workflow apps with same-named workflows — e.g. two instances of the same framework), same
WORKFLOW_POSTGRES_URL, distinctWORKFLOW_QUEUE_NAMESPACE. - Start a long-running workflow in A.
- While it is
running, boot B. B logs[world-postgres] Re-enqueued N active run(s) on startupincluding A's run, under B's namespace. - B's runner claims and executes A's run (same
workflow_flowstask id; handler reconstructs the topic with B's prefix). With distinctWORKFLOW_POSTGRES_JOB_PREFIXvalues the takeover becomes deterministic: the re-enqueued foreign run can only be claimed by B.
Suggested directions
Either would resolve this class of issue:
- Scope claiming and recovery to the world's identity — e.g. include the namespace in the graphile task identifier (or validate the producer namespace in the handler and re-enqueue/skip on mismatch), and give
runs.lista filter thatreenqueueActiveRunspasses through (thedeployment_idcolumn looks like a natural hook, similar in spirit to the world-local tag scoping in #2667). - Or, if a Postgres world database is intended to be exclusive to a single application, state that explicitly in the world-postgres docs — today nothing fails loudly when two applications share one database; they just silently cross-execute.
Workaround
We moved to one Postgres database per application (derived from a base URL, created + workflow-postgres-setup bootstrapped on demand). That fully isolates storage, queue, and recovery, and has been working well — sharing it here for other self-hosters who hit the same symptom.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/world-postgres/src/queue.ts and src/index.ts, then trace recovery in packages/world/src/recovery.ts. Reproduce the two-application scenario with one Postgres database and distinct queue namespaces, comparing claiming with startup recovery. Done means applications sharing a database cannot claim or re-enqueue another application's active runs, or the documentation clearly states that the database must be exclusive.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, typescript
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100