Step failure publication is lost when a step throws a thenable: normalizeUnknownError awaits the raw value unbounded
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Summary
A retryable step-body failure can fail to publish its step_retrying / step_failed event entirely when the thrown VALUE is hostile, leaving queue visibility-timeout redelivery as the only recovery. We hit this in production on @workflow/core@5.0.0-beta.35: a doStreamStep provider stream rejected ~43s into a step with a nameless non-Error value; no step_retrying was ever written and the run sat silent for ~824s until the ownership-lease redelivery started attempt 2.
Two seams between the throw and the durable event both consume the raw thrown value:
1. normalizeUnknownError (dist/types.js) awaits a thrown thenable unbounded
export async function normalizeUnknownError(value) {
if (isThenable(value)) {
try {
const resolved = await value; // <-- unbounded
Two hostile classes:
// (a) never settles — the await parks forever, and with it the step executor's
// catch path and the runtime's run-terminal handler. No step_retrying, no logs.
const pending = { then() {} };
// (b) self-resolving — native promise adoption enqueues a
// PromiseResolveThenableJob that re-enqueues itself forever: an unbounded
// microtask storm. Note a Promise.race timeout CANNOT bound this class —
// microtasks drain to exhaustion before any macrotask, so the timer never
// fires. We verified this with CPU sampling (a vitest worker pinned at 100%
// inside Builtins_PromiseResolveThenableJob for 17 minutes).
const chain = { then(r) { r(chain); } };
Because of (b), the fix cannot merely race the await against a timer. It has to assimilate manually — call .then itself and never hand the value to native promise resolution — with a deadline for the never-settling class, a depth cap for chains (native adoption flattens resolve-chains invisibly, so a JS-side counter around await can never engage), and a never-throw guarantee, since this function IS the failure-publication path.
2. dehydrateStepError runs inside the step_retrying / step_failed publication try blocks
An unserializable thrown graph (e.g. an Error carrying a symbol-valued property) throws from inside the publication path, so a retryable failure escapes the executor and fails the run instead of retrying the step. Bounding the dehydrate and falling back to a minimal always-serializable Error built from the already-normalized {name, message, stack} descriptor closes it.
Repro sketch
Any workflow step whose body ends with throw pending (shape (a) above) never publishes a step terminal event; the step is retried only by queue redelivery minutes later. Shape (b) additionally pins a CPU at 100%.
What we did
We ship both fixes as a pnpm patch in production (manual thenable assimilation with a 1s deadline + depth cap 8 + never-throw in normalizeUnknownError; a bounded dehydrateStepError wrapper with descriptor fallback at the three publication sites) and have a test suite driving the patched dist through all the classes above. Happy to share the patch or open a PR against src/ if useful — the semantics are upstream-neutral.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with normalizeUnknownError in dist/types.js and trace its source implementation under src/. Then inspect the three publication sites where dehydrateStepError runs inside step_retrying or step_failed handling. Run the existing test suite against pending, self-resolving, and unserializable thrown values; done means failure events still publish and hostile values cannot block or escape the failure path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100