Performance Megathread - Is your workflow run slow? Look here first

This issue gathers all known performance issues that are NOT due to edge-cases.

If your workflow performs well in most cases, but hits performance problems in some cases, please open a separate issue, and optionally link it here.

Current bottlenecks

• Every ~2 minutes we create a new invocation to stay ahead of serverless function timeouts. This could be tightened (#1927), will be fixed in #2964.
• Using Promise.all with over 3 steps will still cause a full event log reload for all steps, delaying step processing by ~200ms-2s.
• Passing a step output as the input for a subsequent step still duplicates the payloads. We should de-duplicate this intelligently.
• Fluid cold starts still affect workflows due to large bundlesize. We should reduce this where possible.
• Streams are still slower to flush than we'd like it to be. Could be sped up by using websockets (#2731)
• Creating the VM every replay is expensive for long runs (fixed in #3046)
• Time to first step is ~200ms, but could be reduced to ~100ms for the median case by eliminating the first network requests

Solved bottlenecks (as of 5.0.0-beta.36)

• Cross-region and multi-region deployments on Vercel were slow, which we fixed by introducing region-pinned runs and multi-region support in >=5.0.0-beta.33.
• Between every step, the workflow enqueues an event replay. This means invocation overhead, queue overhead, and initial state fetching overhead on each step call. >=5.0.0-beta.5 removed 90%+ of this overhead (#1338)
• Large network overhead for event log payload fetching was solved in >=5.0.0-beta.16 (#2055)
• DurableAgent/WorkflowAgent excessive step size overhead was fixed in #1929
  • Consider turning off collectUIChunks if your steps are still too large
• Step input/output size was reduced by 80% in #2394
• Large bundlesize due to sourcemaps (#1842) was fixed by defaulting sourcemaps to off
• Stream performance was improved in >=5.0.0-beta.36

A note on pre-releases

Performance improvements go through a QA phase before being released as 5.x..x-beta releases currently. If you want the latest improvements ahead of the beta release, considering using pre-releases (nightly build) by installing tarballs from https://workflow-tarballs.labs.vercel.dev/

Review the Current bottlenecks and Solved bottlenecks sections, then follow the referenced issues such as #1927, #2731, and #2964 to identify one specific performance problem. This megathread names no files or tests and has no single completion condition; a useful contribution would need a separately scoped issue with measurable improvement criteria.