vercel / vercel/workflow

world-postgres: crash recovery runs only at startup, so a stranded run waits for a deploy

Open
#3,993 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Packages: @workflow/world-postgres@4.3.0, @workflow/world@4.2.1
Runtime: AWS ECS Fargate, rolling deploys

What happens

reenqueueActiveRuns is the only mechanism that creates fresh work for a run whose job was lost, and it is called exactly once, inside start():

async start() {
  await queue.start();
  await reenqueueActiveRuns(storage.runs, queue.queue, 'world-postgres');
}

There is no periodic equivalent, and no documented way to invoke the sweep against an already-running world. So a run stranded by a lost job recovers only when some process boots.

Why this matters

On a rolling-deploy platform, "some process boots" means "someone deploys". In a busy environment that looks self-healing; in production, where releases are occasional, a stranded run waits for the next one.

Meanwhile the run stays running in workflow_runs, so anything reading run status believes work is in progress.

Worth noting separately: the sweep reports itself with console.log ("Re-enqueued N active run(s) on startup"). With an OpenTelemetry setup that instruments a logger rather than the console, there is no signal that recovery ran or what it found.

Evidence

A run interrupted by a deploy at 09:53 made no progress for 31 minutes. It resumed within 30 seconds of an unrelated deploy at 10:25, whose boot sweep minted a fresh job. Nothing in between would have recovered it.

What we would like

A periodic recovery pass, or a documented supported way to run the sweep from application code so we can schedule it ourselves.

reenqueueActiveRuns is importable from @workflow/world today, but calling it against a live world is not a documented contract, and it re-enqueues every pending/running run indiscriminately — which is right at boot and too blunt on a timer, since a run that is legitimately mid-step would be re-enqueued alongside the stranded ones.


Related: #3992 (attempts are consumed by lost workers rather than failures, which is what strands the run in the first place) and #3994 (the retry cap is not configurable).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating reenqueueActiveRuns and the world-postgres start() path, then inspect how workflow run states and queue jobs are represented. Review existing tests around startup recovery and the package exports. Done should provide either a supported, documented recovery entry point or a periodic pass that distinguishes stranded runs from legitimate in-progress steps, with coverage for the reported deploy interruption.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, typescript
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.