vercel / vercel/workflow

Step failure publication is lost when a step throws a thenable: normalizeUnknownError awaits the raw value unbounded

Open
#2,994 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Summary

A retryable step-body failure can fail to publish its step_retrying / step_failed event entirely when the thrown VALUE is hostile, leaving queue visibility-timeout redelivery as the only recovery. We hit this in production on @workflow/core@5.0.0-beta.35: a doStreamStep provider stream rejected ~43s into a step with a nameless non-Error value; no step_retrying was ever written and the run sat silent for ~824s until the ownership-lease redelivery started attempt 2.

Two seams between the throw and the durable event both consume the raw thrown value:

1. normalizeUnknownError (dist/types.js) awaits a thrown thenable unbounded
export async function normalizeUnknownError(value) {
    if (isThenable(value)) {
        try {
            const resolved = await value;   // <-- unbounded

Two hostile classes:

// (a) never settles — the await parks forever, and with it the step executor's
// catch path and the runtime's run-terminal handler. No step_retrying, no logs.
const pending = { then() {} };

// (b) self-resolving — native promise adoption enqueues a
// PromiseResolveThenableJob that re-enqueues itself forever: an unbounded
// microtask storm. Note a Promise.race timeout CANNOT bound this class —
// microtasks drain to exhaustion before any macrotask, so the timer never
// fires. We verified this with CPU sampling (a vitest worker pinned at 100%
// inside Builtins_PromiseResolveThenableJob for 17 minutes).
const chain = { then(r) { r(chain); } };

Because of (b), the fix cannot merely race the await against a timer. It has to assimilate manually — call .then itself and never hand the value to native promise resolution — with a deadline for the never-settling class, a depth cap for chains (native adoption flattens resolve-chains invisibly, so a JS-side counter around await can never engage), and a never-throw guarantee, since this function IS the failure-publication path.

2. dehydrateStepError runs inside the step_retrying / step_failed publication try blocks

An unserializable thrown graph (e.g. an Error carrying a symbol-valued property) throws from inside the publication path, so a retryable failure escapes the executor and fails the run instead of retrying the step. Bounding the dehydrate and falling back to a minimal always-serializable Error built from the already-normalized {name, message, stack} descriptor closes it.

Repro sketch

Any workflow step whose body ends with throw pending (shape (a) above) never publishes a step terminal event; the step is retried only by queue redelivery minutes later. Shape (b) additionally pins a CPU at 100%.

What we did

We ship both fixes as a pnpm patch in production (manual thenable assimilation with a 1s deadline + depth cap 8 + never-throw in normalizeUnknownError; a bounded dehydrateStepError wrapper with descriptor fallback at the three publication sites) and have a test suite driving the patched dist through all the classes above. Happy to share the patch or open a PR against src/ if useful — the semantics are upstream-neutral.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with normalizeUnknownError in dist/types.js and trace its source implementation under src/. Then inspect the three publication sites where dehydrateStepError runs inside step_retrying or step_failed handling. Run the existing test suite against pending, self-resolving, and unserializable thrown values; done means failure events still publish and hostile values cannot block or escape the failure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.