temporalio / temporalio/temporal

Support data references to avoid storing the same Payload multiple times

Open
#4,502 1 comment 9 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
The same value can be an argument of many activities. For example, one activity returns a document, and then 1k activities are invoked, each of them receiving a document as one of the arguments. In this case, the document is going to be stored 1k+1 times in the history which is wasteful.

Describe the solution you'd like
My strawman is to introduce a special command (or reuse Marker command) that would store a value (as Payloads) in the history. Add support for activity/child workflow inputs and outputs reference that value (by eventId) instead of including it directly into the corresponding schedule commands.

Another option is to support referencing Payloads directly inside other events like WorkflowExecutionStarted.

Additional context
From Slack:

To make this more concrete, i'm implementing a DSL on top of a Temporal workflow.
The DSL workflow definition has some things that need to be extracted and consumed throughout, sometimes.
Currently i'm passing around my DSL string to all functions as a "dsl workflow context"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by studying the proposed Marker command, Payload storage, activity and child workflow input/output references, schedule commands, and WorkflowExecutionStarted events described in the issue. Define the event and history semantics before implementation; done would include a decided reference model that avoids duplicate Payload storage while preserving workflow behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.