caolan / caolan/highland

async mapping?

Open
#313 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3.4k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Having trouble determining how to best read and async map, from a file with a consumer that pauses/resumes the stream.

My first attempt was something like the following:

```
var mapper = function (raw) {
//...parsing was happening here
return result;
};

var consumer = function (err, x, push, next) {
//... if not err or nil, write batch to db
// there is some detection here to pause and resume the stream to only have a few concurrent writes to the db at a time.
};

var stream = _(fs.createReadStream(filepath)).split().map(mapper).batch(3000).consume(consumer);
```

This was working fine, and it was throttling with the pause resume with no problems. However, due to some changes i'm required to have the mapper parsing each row asynchronously, but i can't seem to figure out what I need to do to make that work with highland.

I tried changing mapper to

```
var mapper = _.wrapCallback(function (raw, cb) {
//... async stuff,
cb(null, result);
});
```

but after the first 500 items, i end up with the script spitting out a bunch of
`(node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral.` until max callstack exceeded.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.