Use external storage for large data attributes [Large Feature]
- Dominant language
- Go
- Stars
- 661
- Forks
- 63
- PR merge metrics
- No merged PRs in 30d
Description
When exceeding the threshold (configurable, e.g. 100KB) for a data attribute, iWF can use others like S3 for storing the data attributes instead of writing into Temporal history. (only storing the keys and the S3 objectIDs)
By storing keys and S3 objectIDs in Temporal history, IWF server will load from S3 before sending to application, and write to S3 for updates. For optimization, server could also load from S3 lazily when application tried to read it.
This is possible because iWF server workflow never really read the value of the DAs -- they are transparent to iWF server.
By offloading the large data attributes to S3, it's much easier for users to deal with large datasets, and more cost effective on using Cadence/Temporal.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by tracing data-attribute persistence and the boundary between the iWF server and applications; clarify the storage-provider, threshold, lazy-loading, and update requirements. Done should include agreed behavior and tests for the external-storage lifecycle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100