Memory usage issue with stream-transform
Open
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 4.3k
- Forks
- 299
- Avg merge
- 16h 19m
- Merged PRs (30d)
- 1
Description
Describe the bug
When using stream-transform for processing large datasets and the parallel option is a value greater than 1, we're seeing high memory usage.
To Reproduce
const fs = require('fs')
const memwatch = require('@airbnb/node-memwatch')
const { pipeline } = require('stream/promises')
const { transform } = require('stream-transform')
let maxUsedHeap = 0
async function main() {
memwatch.on('stats', (stats) => {
maxUsedHeap = Math.max(maxUsedHeap, stats.used_heap_size)
})
await pipeline(
function* () {
let i = -1
const n = 9999999
while (++i < n) {
yield { i }
}
},
transform({ parallel: +process.env.PARALLEL }, (chunk, next) =>
next(null, chunk.i)
),
fs.createWriteStream('/tmp/output')
)
console.log(`${maxUsedHeap / (1000 * 1000)}mb`)
}
main()
// $ PARALLEL=1 node example.js
// 6.009856mb
// $ PARALLEL=2 node example.js
// 320.684144mb
Additional context
- Our theory is that this is backpressure-related. We noticed that
this.pushwas returningfalseto indicate that the stream should pause reading, yet stream-transform asks for more input over here regardless. - To support that theory, this seems to resolve the issue, though it isn't a proper solution: https://gist.github.com/477d30dfeb443be9a92ac8a3aedc238f
- Thank you for the CSV project :) its helping us a ton here at snaplet.dev
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/stream-transform/lib/index.js, especially the input request around line 40 and the this.push call around line 85. Run the supplied large-dataset reproduction with PARALLEL=1 and PARALLEL=2, then inspect the linked gist for comparison. Done means parallel processing respects backpressure without the reported memory growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100