Runtime: allow streams to be "serialized" to state internally
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 21
- Forks
- 23
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 17
Description
We need to think about how some special data structures, like streams, can be written to state and shared between steps in the same workflow.
Usually right now we strictly serialize state objects which can cause loss of data (regexes will be ruined for example, and functions dropped).
But internally, it's super useful to be able to share a stream between steps. Then eg the http adaptor can fetch a stream from the server, and pass it to eg dhis2 to upload directly from the stream.
This may also relate to #648, if the same solution allows functions to be internally written to state.
There are two things to watch out for:
- The final state returned by the runtime must be safely serialisable. A lot of workflows don't care about the notion of "final state", but some do, and we generally need to make sure we have consistent handling
- The worker will send dataclip output from each step to lightning through a websocket. This needs to be properly serializable so that it can be sent to lightning. That does raise a problem of: if the input to a step requires a non-serializable value (like a stream), how could we resume the run from that point? We might need some means of flagging to say "this step is not reproducible". Same problem in the CLI.
So maybe we have a serialize() function somewhere in util with a flag for internal and external mode? Or strict and non-strict?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing state handling in the runtime, then inspect how the worker sends dataclip output over its websocket and how the CLI handles resuming runs. Define the internal versus external serialization behavior, including final state and non-reproducible steps, before identifying the relevant implementation and tests; done means streams can be shared internally while externally returned and transmitted state remains safely serializable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100