vercel / vercel/workflow

Owner-aware / single-flight boot recovery for multi-instance self-hosted deployments

Open
#3,138 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Follow-up from #2544 (boot-time recovery via ensureWorldStarted()).

Problem

start()'s boot recovery re-enqueues every pending/running run in storage — it does not know which instance is responsible for a run. On a horizontally-scaled or rolling deployment of the Postgres World, every booting instance therefore performs a full paginated scan of the active runs and re-enqueues all of them: N replicas ⇒ N scans + N duplicate enqueues per in-flight run on every deploy. Replay idempotency, the inline-ownership lease, and job-key dedup make this correct, but at scale it is a thundering herd of redundant scans, enqueues, and wake replays.

Current mitigation

WORKFLOW_SKIP_BOOT_RECOVERY=1 (added in #2544) lets operators run recovery from a single designated instance and opt the rest out. That is manual designation, not a mechanism.

Possible directions

  • Single-flight: an advisory lock (e.g. pg_advisory_lock) or a recovery-epoch row so only one booting instance per deploy performs the scan.
  • Owner-aware: attribute runs to instances (lease/heartbeat) so a booting instance only re-drives runs whose owner is gone; this would also shorten the mid-step recovery delay currently bounded by the inline-ownership lease (WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS, default 860s), since a dead owner could be detected affirmatively instead of waiting out the lease.

Related: #2978 (shared-DB namespace isolation for recovery), #2780 (inline-ownership lease).

🤖 Generated with Claude Code

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with start() and ensureWorldStarted(), then read the context in #2544, #2978, and #2780. Compare the proposed single-flight and owner-aware directions; done should prevent redundant boot recovery scans and duplicate enqueues for multi-instance deployments.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.