world-postgres: crash recovery runs only at startup, so a stranded run waits for a deploy
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Packages: @workflow/world-postgres@4.3.0, @workflow/world@4.2.1
Runtime: AWS ECS Fargate, rolling deploys
What happens
reenqueueActiveRuns is the only mechanism that creates fresh work for a run whose job was lost, and it is called exactly once, inside start():
async start() {
await queue.start();
await reenqueueActiveRuns(storage.runs, queue.queue, 'world-postgres');
}
There is no periodic equivalent, and no documented way to invoke the sweep against an already-running world. So a run stranded by a lost job recovers only when some process boots.
Why this matters
On a rolling-deploy platform, "some process boots" means "someone deploys". In a busy environment that looks self-healing; in production, where releases are occasional, a stranded run waits for the next one.
Meanwhile the run stays running in workflow_runs, so anything reading run status believes work is in progress.
Worth noting separately: the sweep reports itself with console.log ("Re-enqueued N active run(s) on startup"). With an OpenTelemetry setup that instruments a logger rather than the console, there is no signal that recovery ran or what it found.
Evidence
A run interrupted by a deploy at 09:53 made no progress for 31 minutes. It resumed within 30 seconds of an unrelated deploy at 10:25, whose boot sweep minted a fresh job. Nothing in between would have recovered it.
What we would like
A periodic recovery pass, or a documented supported way to run the sweep from application code so we can schedule it ourselves.
reenqueueActiveRuns is importable from @workflow/world today, but calling it against a live world is not a documented contract, and it re-enqueues every pending/running run indiscriminately — which is right at boot and too blunt on a timer, since a run that is legitimately mid-step would be re-enqueued alongside the stranded ones.
Related: #3992 (attempts are consumed by lost workers rather than failures, which is what strands the run in the first place) and #3994 (the retry cap is not configurable).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating reenqueueActiveRuns and the world-postgres start() path, then inspect how workflow run states and queue jobs are represented. Review existing tests around startup recovery and the package exports. Done should provide either a supported, documented recovery entry point or a periodic pass that distinguishes stranded runs from legitimate in-progress steps, with coverage for the reported deploy interruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, typescript
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100