indeedeng / indeedeng/iwf

Use external storage for large data attributes [Large Feature]

Open
#506 4 comments 0 reactions 0 assignees View on GitHub
large
Dominant language
Go
Stars
661
Forks
63
PR merge metrics
No merged PRs in 30d

Description

When exceeding the threshold (configurable, e.g. 100KB) for a data attribute, iWF can use others like S3 for storing the data attributes instead of writing into Temporal history. (only storing the keys and the S3 objectIDs)

By storing keys and S3 objectIDs in Temporal history, IWF server will load from S3 before sending to application, and write to S3 for updates. For optimization, server could also load from S3 lazily when application tried to read it.

This is possible because iWF server workflow never really read the value of the DAs -- they are transparent to iWF server.

By offloading the large data attributes to S3, it's much easier for users to deal with large datasets, and more cost effective on using Cadence/Temporal.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by tracing data-attribute persistence and the boundary between the iWF server and applications; clarify the storage-provider, threshold, lazy-loading, and update requirements. Done should include agreed behavior and tests for the external-storage lifecycle.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.