temporalio / temporalio/cli

[Feature Request] Decode workflow histories concurrently with remote codec servers

Open
#1,193 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
379
Forks
103
Avg merge
2d 11h
Merged PRs (30d)
23

Description

Is your feature request related to a problem? Please describe.

When the CLI is using Temporal Cloud and configured with a remote codec server, exporting a large workflow history can be very slow:

temporal workflow show \
  --workflow-id <workflow-id> \
  --output json > history.json

The codec protocol supports sending multiple payloads in one /decode request. However, the CLI's payload visitor makes a separate request for each payload or Payloads field it encounters and processes those visits serially.

Because workflows contain so many payloads, even a small amount of latency (~50ms) causes some exports to take 5+ minutes for us.

Increasing the RPC timeout prevents the operation from timing out, but does not address the latency.

Describe the solution you'd like

Remote codec calls made while processing a workflow history should be concurrent, batched more broadly, or otherwise avoid paying one network round trip serially for every visited field.

api-go now supports opt-in concurrent payload visiting through VisitPayloadsOptions.ConcurrencyLimit, which may provide a path to doing this with bounded concurrency.

Additional context

This was previously recognized in:

That issue explicitly mentioned concurrently calling the codec instead of using the serial visitor. It was closed by https://github.com/temporalio/cli/pull/676, which added a configurable RPC timeout but did not address the underlying latency.

There is also an open cross-SDK feature tracking concurrent payload visiting:

The Temporal CLI item remains unchecked. The underlying Go support has since landed:

The CLI currently constructs NewPayloadCodecGRPCClientInterceptor here:

https://github.com/temporalio/cli/blob/main/cliext/client.go

However, that interceptor is currently used without configuring the payload visitor's concurrency option.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in cliext/client.go, where NewPayloadCodecGRPCClientInterceptor is constructed, and read the api-go VisitPayloadsOptions.ConcurrencyLimit support. Trace how the interceptor's payload visitor processes workflow history payloads and determine how bounded concurrency should be configured. Done means remote codec requests no longer wait serially for every visited field while preserving the CLI's workflow history export behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.