OpenFn / OpenFn/kit

Runtime: allow streams to be "serialized" to state internally

Open
#865 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

DevX
Dominant language
TypeScript
Stars
21
Forks
23
Avg merge
1d 22h
Merged PRs (30d)
17

Description

We need to think about how some special data structures, like streams, can be written to state and shared between steps in the same workflow.

Usually right now we strictly serialize state objects which can cause loss of data (regexes will be ruined for example, and functions dropped).

But internally, it's super useful to be able to share a stream between steps. Then eg the http adaptor can fetch a stream from the server, and pass it to eg dhis2 to upload directly from the stream.

This may also relate to #648, if the same solution allows functions to be internally written to state.

There are two things to watch out for:

  • The final state returned by the runtime must be safely serialisable. A lot of workflows don't care about the notion of "final state", but some do, and we generally need to make sure we have consistent handling
  • The worker will send dataclip output from each step to lightning through a websocket. This needs to be properly serializable so that it can be sent to lightning. That does raise a problem of: if the input to a step requires a non-serializable value (like a stream), how could we resume the run from that point? We might need some means of flagging to say "this step is not reproducible". Same problem in the CLI.

So maybe we have a serialize() function somewhere in util with a flag for internal and external mode? Or strict and non-strict?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing state handling in the runtime, then inspect how the worker sends dataclip output over its websocket and how the CLI handles resuming runs. Define the internal versus external serialization behavior, including final state and non-reproducible steps, before identifying the relevant implementation and tests; done means streams can be shared internally while externally returned and transmitted state remains safely serializable.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.