PipedreamHQ / PipedreamHQ/pipedream

[FEATURE] Step Memoisation

Open
#806 0 comments 0 reactions 0 assignees View on GitHub
enhancement triaged
Dominant language
JavaScript
Stars
11.7k
Forks
5.8k
Avg merge
3d 10h
Merged PRs (30d)
102

Description

**Is your feature request related to a problem? Please describe.**

As we break down logic into individual steps, there are times we need to interact with another system in a non-idempotent way (side affect / mutation). Rerunning old events to test code fixes can become cumbersome as "skipping" steps isn't trivial, especially when they output something needed further down the chain.

**Describe the solution you'd like**

The ability to mark a step to memoise. The inputs for the step should be calculate based on the code, not the entire event chain.

IE, the memoisation key would be a composite key of the values of any `event.`, `param.`, or `step.` in the steps code.

When the calculated inputs have been seen before, the exports and return value can be replayed.

**Do you have a workaround?**

Conditional checkpoint logic

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.