caolan / caolan/highland

highland and workflow processing

Open
#122 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3.4k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Hello,

I am trying to use highland for a streaming workflow processor and thought that it would maybe be interesting to discuss it here to see how the highland primitives can be used for this (I hope this is the right place). I have :

```
var wfs = [wf1, wf2, wf3,...]
```

where wfN are objects representing different workflows.

each workflow can advance one step by calling `wfN.walk()` which is a promise for the end of the current step. it is resolved to wfN.

so I understand that

```
var walk = function(wf) { return _(wf.walk()) }
var s = _(wfs).flatMap(walk).resume()
```

will make all the workflows advance by 1 step, and with

```
var s = _(wfs).flatMap(walk).flatMap(walk).resume()
```

I can have the workflows advance 2 steps.

the `wf.walk()` is stable when the workflow is finished so I can call it many times.

```
var s = _(wfs).flatMap(walk).(..N times..).flatMap(walk).resume()
```

would walk N steps and finish all workflows if they have less than N steps.

Now the thing is that each workflow has a different number ot steps to execute so I don't know how to optimize the setup of the stream graph without making an hypothesis on the maximum number of steps required.

I can also use something like

```
var walk = function(wf) {
if (wf.finished()) return _([wf])
return _(wf.walk()).map(walk)
}
var s = _(wfs).flatMap(walk).resume()
```

in order to build a sort of recursive stream graph but it feels non natural and maybe not optimal. I also have a hard time seeing what happens with this if I call

```
var s = _(wfs).flatMap(walk).parallel(10).resume()
```

I hope my use case is correctly described. Do you see a problem in this approach and is there a better `highland` way of doing something like that ?

thanks for making highland, it is a very inspiring way of doing things in node

Contributor guide

Open the contributing guide

Research direction

The issue names no repository file, test, or entry point. Start by understanding the demonstrated flatMap, recursive walk, and parallel patterns, then determine whether the request needs an API change, documentation, or only guidance; done requires a maintainer-approved, explicitly scoped outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.