Option to bound memory usage
- Dominant language
- JavaScript
- Stars
- 419
- Forks
- 55
- PR merge metrics
- No merged PRs in 30d
Description
_(Please correct me if I'm wrong with any of the below. I love this library, but I'll cut to it...)_
`decompress` currently crashes the process with `137` if the sum of the input file and its decompressed contents exceeds the available memory of the system.
This is because the [entire input file is loaded into memory](https://github.com/kevva/decompress/blob/84a8c1046946add1a6ae01c54dbebf312e4ffc85/index.js#L148) and [all extracted files are also retained in memory](https://github.com/kevva/decompress-unzip/blob/0c33871268e2eee3cb5370bec8ae8a82d887f5d7/index.js#L44).
As a result, you can only decompress files where the file itself _summed with_ its decompressed contents is less than the available memory of your system (`Buffer`s come from available memory, not from V8 heap, so aren't limited by `--max-old-space-size`, but are of course still limited by the amount of available memory the system has).
This is fine on development machines, or very large servers, but even on a `medium` EC2 instance that has 1.5GB of available memory, you're limited to extract files under 1.5GB (this includes their extracted content). On a `nano` instance you can only decompress files under 30MB before you get a 137 exit code (depending on the compression ratio) 😅
_- breathes -_
Do you see a future where `decompress` uses streams instead? (For backward compatibility you could always then buffer everything from the streams -- but at least users would have the option to opt-out of the "buffer everything in memory" behaviour if needed.)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with index.js around the linked input-loading code and decompress-unzip/index.js around the linked extracted-file handling. Trace how buffers move through both packages, then define a streaming or opt-out path that avoids retaining all data while preserving existing behavior and verify it with the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100