Step arguments fail to decrypt when a run spans two deployments: step handler resolves encryption key from runId only, and the error is silently swallowed by the queue retry callback
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Summary
When a workflow run is started by one deployment and its steps execute on another (common on preview branches,
where superseded deployments are deactivated and queue callbacks re-route to the branch's latest deployment),
every step attempt fails before user code runs with:
RuntimeDecryptionError: Encrypted data encountered but no encryption key is available.
Encryption is not configured or no key was provided for this run.
The run then fails as FatalError: Step "..." exceeded max retries (N retries) with a blank underlying stack,
because the per-attempt error is dropped (see "Why it's invisible" below). User-level try/catch inside the step
never executes.
Root cause
Two asymmetric key resolutions in @workflow/core 4.6.0:
- Workflow handler (dist/runtime.js, replay path) calls world.getEncryptionKeyForRun(workflowRun) with the run
object. In @workflow/world-vercel (dist/encryption.js), run.deploymentId differs from VERCEL_DEPLOYMENT_ID, so
it fetches the origin deployment's key from the Vercel API. Replay works and step inputs get
dehydrated/encrypted with the origin deployment's run key. - Step handler (dist/runtime/step-handler.js:356) calls world.getEncryptionKeyForRun?.(workflowRunId) with only
the string. With no deploymentId context, the Vercel world takes the local-derivation branch
(isServerlessRuntime && !deploymentId) and derives from the current deployment's VERCEL_DEPLOYMENT_KEY — the
wrong key material for a cross-deployment run (or none). Hydrating the encrypted step input then throws
RuntimeDecryptionError before stepFn.apply().
Still present in 5.0.0-beta.27: memoizeEncryptionKey(world, workflowRunId) is called with the bare string in
the beta step handler.
Why it's invisible
The pre-user-code error propagates to the queue handler, whose retry directive receives it and discards it
(@workflow/world-vercel/dist/queue.js):
retry: (_error, { deliveryCount }) => ({
afterSeconds: getHandlerErrorRetryAfterSeconds(deliveryCount),
}),
Nothing logs _error. The runtime's info/debug logs are also console-silent unless DEBUG=workflow:* is set, so
the operator sees only the final exceeded max retries with stack: step.error?.stack = undefined (no attempt
ever recorded a step error). Diagnosing this required patching the retry callback to print the swallowed error.
Reproduction
- Next.js app on Vercel, workflow 4.6.0, on a git branch with a custom environment/domain (probably plain
preview too). - Load the app, then push a new deployment of the branch.
- Without reloading, trigger a workflow (skew protection routes the POST to the old deployment; queue
callbacks execute on the new one). - First step of the run exhausts retries with the blank exceeded max retries; per-attempt
RuntimeDecryptionError only visible with the logging patch.
Suggested fixes
- In the step handler, resolve the key deployment-aware, e.g. fetch the run (world.runs.get(runId, {
resolveData: 'none' })) and pass it to getEncryptionKeyForRun, matching the workflow handler. We've verified
this patch fixes the failure in our app. - Log the error in the queue retry callback (or record it as a step attempt error) so retry-exhaustion is
diagnosable.
Environment
- workflow 4.6.0 / @workflow/core 4.6.0 / @workflow/world-vercel 4.5.0 / @workflow/ai 4.1.2
- Next.js 16.0.10 (Turbopack), Node 24, Vercel Fluid Compute, iad1
- Run IDs available privately on request.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in dist/runtime/step-handler.js at line 356 and compare its encryption-key lookup with the workflow handler in dist/runtime.js. Read @workflow/world-vercel/dist/encryption.js and dist/queue.js, then reproduce the cross-deployment retry path described in the issue. Done means step inputs decrypt across deployments and the retry path preserves enough error information to diagnose exhaustion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- next.js, node.js, typescript
- Domain
- backend, distributed-systems, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100