temporalio / temporalio/temporal

Optimize large payload fan-out to activities and child workflows

Open
#1,492 1 comment 12 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

devexp enhancement up-for-grabs
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
Some use cases need to pass the same payload to a large number of activities or child worklfows. In the current model, the payload will be repeated in the history as many times as the number of invocations. Recording the same payload many times is clearly nonoptimal.

Describe the solution you'd like
I don't have a specific API/feature proposal to resolve this issue. But I believe some solution that saves such a payload only once should be possible.

One simplistic strawman is to have a special command to save a payload and then allow referencing it in other places of a workflow.

Another idea is to perform payload deduplication by storing large payload hashes in the mutable state.

Additional context
A forum post that describes a customer having issues with a large payload passed to many child workflows.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the issue and the linked forum post about workflow history size limits. No implementation files, tests, entry points, or concrete API are identified; the work is not ready to pick up until a design is agreed for storing a large payload once and referencing or deduplicating it across activity and child-workflow invocations.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.