apple / apple/swift-temporal-sdk

[Feature Discussion] External Storage

Open
#157 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
315
Forks
31
Avg merge
4d 11h
Merged PRs (30d)
4

Description

Hey folks,

I'm on the Temporal SDK team and I'm currently implementing [External Storage](https://docs.temporal.io/external-storage) in the Temporal [TypeScript and Java SDKs](https://github.com/temporalio/sdk-typescript/pull/2092) and I'd like to gauge interest in bringing it to the Swift SDK. I'd be happy to put together a high-level design for review that aligns with the other SDKs. I would also have some time over the next few months to contribute PRs.

For context, External Storage offloads large payloads to providers like S3, GCS, or Azure so they don't take up space in your workflow history. Instead of storing the large payload, the history only contains a small claim check. Workers use these claim checks to retrieve the payloads on-demand. As for level of effort, there would be roughly 6-7 units of work for an initial release:
- Pull in new proto types (reference payload) and create base extstore types.
- Implement a payload visitor. Need a way to walk proto messages to find payloads.
- DataConverter wiring. run extstore after codec on encode, before codec on decode.
- StorageDriver interface and built-in drivers. store/retrieve interface, content-addressed keys, hashing, integrity check, cancellation, etc
- Reference payload caching. cache previously stored/retrieved references
- Config and metrics. size thresholds, drivers selector, driver registry

For reference, [here is the Python External Storage feature](https://docs.temporal.io/develop/python/data-handling/external-storage) in Public Preview.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Temporal External Storage documentation and comparing the TypeScript/Java PR 2092 and Python feature. Then review the six to seven proposed work areas: proto types, payload visiting, DataConverter wiring, storage drivers, caching, configuration, and metrics. Done means an agreed design and an initial Swift SDK release covering the selected scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.