vercel / vercel/workflow

TTLS: fan-out completion checks are O(N²) event-log loads — return a pending-step count on step-terminal writes

Open
#3,455 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Problem

At high fan-out, TTLS (time to last step) falls off a cliff: durabench parallel sweeps measure ~15–20s TTLS at 256 branches vs ~1.3s at 64, on every SDK version (node engine, iad1). Trace decomposition of a representative run (wrun_41KZR0MW890GWZK34RD4Y1JBDT) shows the dominant cause is O(N²) event-log read traffic:

  • Every queued step's consumer, after completing its step, loads the run's full event log to answer "are all parallel steps done?" (the last completer replays inline). At N branches that is N loads of an ~4N-event log.
  • Measured server-side: GET /api/v4/runs/:runId/events p90 = 7.9s during the burst; the saturation drags the consumer prologue's runs.get to p90 = 5.1s, smearing branch starts over ~10–15s.
  • Not throttling: zero 425/429 on the run.

Proposal

Let the step-terminal write answer the question the events.list is asking. The World's events.create for step_completed / step_failed returns an optional count of the run's non-terminal steps after the write (or a boolean "none pending"); the runtime then:

  • skips the events.list entirely when pending > 0 (return, another handler finishes later);
  • only the true last completer (pending === 0) proceeds — and only IT loads the log for the inline replay.

That converts O(N) full-log loads per fan-out into exactly one.

World interface impact

Additive and backward-compatible, following the sinceCursor / inline-delta precedent:

  • CreateEventParams: opt-in request flag (e.g. pendingStepCount: true) so backends that must do extra work only do it when asked.
  • EventResult: optional pendingStepCount?: number on step-terminal writes. Absent → runtime falls back to today's events.list check, so old/unsupporting Worlds are unaffected.
  • world-local / world-postgres: cheap direct count. The Vercel backend can maintain a pending counter on the run row (it already patches the step row in the same write) — tracked separately server-side.

Measurement

Baseline and after: durabench parallel sweep at branches {64, 256, 1024}, plus the pnpm phases decomposition (event-log-loads phase should collapse). Expected: the 256-branch TTLS cliff drops toward the queue-transit floor.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the World's events.create path and the runtime's step-terminal handling, then inspect CreateEventParams and EventResult plus the world-local and world-postgres implementations. Run the durabench parallel sweep and pnpm phases; done means supported writes return pending-step information, non-last completers avoid events.list, unsupported Worlds retain the fallback, and event-log loads collapse.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.