Which of these methods to limit how fast things get piped?
- Dominant language
- JavaScript
- Stars
- 3.4k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
In the example below, I'm downloading a gzipped file that's several hundred megabytes (millions of rows), then piping it through highland so I can use the `batch` feature. This way, instead of inserting 1 row into the db at a time (csvStream emits one 'data' event for one row), I can do it for 10000 rows at a time in the 'data' event handler you see below.
```
download.stream(url)
.pipe(zlib.createGunzip())
.pipe(csvStream)
.pipe(highland())
.batch(10000)
.on('data', async (data) => {
await insertIntoDB(data)
})
```
But when running this I get out of memory errors and the system starts to slow down significantly. I think it's because the data is coming in too fast to the 'data' event. The csvStream's `finish` event happens in a couple of minutes but the program runs for up to another hour, which indicates that the whole csv file has been read into memory, rather than being piped downstream piece by piece as the `data` event consumes the batches.
I'm new to highland and looking through the documentation I can't tell which of the various methods would be the most appropriate in this case. http://highlandjs.org/#backpressure seems like it's most relevant to this situation but I can't tell how to use it in this code. http://highlandjs.org/#parallel looks good too.
Can I configure highland so that at any time there's only for example 3 batches (where `batch` is 10000) worth of rows that have been read? And it only reads another 10000 rows when one of those 3 batches is complete.
Contributor guide
Assessment
This issue has not been assessed yet.