highland and workflow processing
- Dominant language
- JavaScript
- Stars
- 3.4k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I am trying to use highland for a streaming workflow processor and thought that it would maybe be interesting to discuss it here to see how the highland primitives can be used for this (I hope this is the right place). I have :
```
var wfs = [wf1, wf2, wf3,...]
```
where wfN are objects representing different workflows.
each workflow can advance one step by calling `wfN.walk()` which is a promise for the end of the current step. it is resolved to wfN.
so I understand that
```
var walk = function(wf) { return _(wf.walk()) }
var s = _(wfs).flatMap(walk).resume()
```
will make all the workflows advance by 1 step, and with
```
var s = _(wfs).flatMap(walk).flatMap(walk).resume()
```
I can have the workflows advance 2 steps.
the `wf.walk()` is stable when the workflow is finished so I can call it many times.
```
var s = _(wfs).flatMap(walk).(..N times..).flatMap(walk).resume()
```
would walk N steps and finish all workflows if they have less than N steps.
Now the thing is that each workflow has a different number ot steps to execute so I don't know how to optimize the setup of the stream graph without making an hypothesis on the maximum number of steps required.
I can also use something like
```
var walk = function(wf) {
if (wf.finished()) return _([wf])
return _(wf.walk()).map(walk)
}
var s = _(wfs).flatMap(walk).resume()
```
in order to build a sort of recursive stream graph but it feels non natural and maybe not optimal. I also have a hard time seeing what happens with this if I call
```
var s = _(wfs).flatMap(walk).parallel(10).resume()
```
I hope my use case is correctly described. Do you see a problem in this approach and is there a better `highland` way of doing something like that ?
thanks for making highland, it is a very inspiring way of doing things in node
Contributor guide
Research direction
The issue names no repository file, test, or entry point. Start by understanding the demonstrated flatMap, recursive walk, and parallel patterns, then determine whether the request needs an API change, documentation, or only guidance; done requires a maintainer-approved, explicitly scoped outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100