vercel / vercel/workflow

Step arguments fail to decrypt when a run spans two deployments: step handler resolves encryption key from runId only, and the error is silently swallowed by the queue retry callback

Open
#2,816 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.4k
Forks
365
Avg merge
2d 11h
Merged PRs (30d)
169

Description

Summary

When a workflow run is started by one deployment and its steps execute on another (common on preview branches,
where superseded deployments are deactivated and queue callbacks re-route to the branch's latest deployment),
every step attempt fails before user code runs with:

RuntimeDecryptionError: Encrypted data encountered but no encryption key is available.
Encryption is not configured or no key was provided for this run.

The run then fails as FatalError: Step "..." exceeded max retries (N retries) with a blank underlying stack,
because the per-attempt error is dropped (see "Why it's invisible" below). User-level try/catch inside the step
never executes.

Root cause

Two asymmetric key resolutions in @workflow/core 4.6.0:

  • Workflow handler (dist/runtime.js, replay path) calls world.getEncryptionKeyForRun(workflowRun) with the run
    object. In @workflow/world-vercel (dist/encryption.js), run.deploymentId differs from VERCEL_DEPLOYMENT_ID, so
    it fetches the origin deployment's key from the Vercel API. Replay works and step inputs get
    dehydrated/encrypted with the origin deployment's run key.
  • Step handler (dist/runtime/step-handler.js:356) calls world.getEncryptionKeyForRun?.(workflowRunId) with only
    the string. With no deploymentId context, the Vercel world takes the local-derivation branch
    (isServerlessRuntime && !deploymentId) and derives from the current deployment's VERCEL_DEPLOYMENT_KEY — the
    wrong key material for a cross-deployment run (or none). Hydrating the encrypted step input then throws
    RuntimeDecryptionError before stepFn.apply().

Still present in 5.0.0-beta.27: memoizeEncryptionKey(world, workflowRunId) is called with the bare string in
the beta step handler.

Why it's invisible

The pre-user-code error propagates to the queue handler, whose retry directive receives it and discards it
(@workflow/world-vercel/dist/queue.js):

retry: (_error, { deliveryCount }) => ({
afterSeconds: getHandlerErrorRetryAfterSeconds(deliveryCount),
}),

Nothing logs _error. The runtime's info/debug logs are also console-silent unless DEBUG=workflow:* is set, so
the operator sees only the final exceeded max retries with stack: step.error?.stack = undefined (no attempt
ever recorded a step error). Diagnosing this required patching the retry callback to print the swallowed error.

Reproduction

  1. Next.js app on Vercel, workflow 4.6.0, on a git branch with a custom environment/domain (probably plain
    preview too).
  2. Load the app, then push a new deployment of the branch.
  3. Without reloading, trigger a workflow (skew protection routes the POST to the old deployment; queue
    callbacks execute on the new one).
  4. First step of the run exhausts retries with the blank exceeded max retries; per-attempt
    RuntimeDecryptionError only visible with the logging patch.

Suggested fixes

  1. In the step handler, resolve the key deployment-aware, e.g. fetch the run (world.runs.get(runId, {
    resolveData: 'none' })) and pass it to getEncryptionKeyForRun, matching the workflow handler. We've verified
    this patch fixes the failure in our app.
  2. Log the error in the queue retry callback (or record it as a step attempt error) so retry-exhaustion is
    diagnosable.

Environment

  • workflow 4.6.0 / @workflow/core 4.6.0 / @workflow/world-vercel 4.5.0 / @workflow/ai 4.1.2
  • Next.js 16.0.10 (Turbopack), Node 24, Vercel Fluid Compute, iad1
  • Run IDs available privately on request.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dist/runtime/step-handler.js at line 356 and compare its encryption-key lookup with the workflow handler in dist/runtime.js. Read @workflow/world-vercel/dist/encryption.js and dist/queue.js, then reproduce the cross-deployment retry path described in the issue. Done means step inputs decrypt across deployments and the retry path preserves enough error information to diagnose exhaustion.

Written by the indexing model from the issue text.

Assessment

Tech stack
next.js, node.js, typescript
Domain
backend, distributed-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.