kevva / kevva/decompress

Option to bound memory usage

Open
#91 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
419
Forks
55
PR merge metrics
No merged PRs in 30d

Description

_(Please correct me if I'm wrong with any of the below. I love this library, but I'll cut to it...)_

`decompress` currently crashes the process with `137` if the sum of the input file and its decompressed contents exceeds the available memory of the system.

This is because the [entire input file is loaded into memory](https://github.com/kevva/decompress/blob/84a8c1046946add1a6ae01c54dbebf312e4ffc85/index.js#L148) and [all extracted files are also retained in memory](https://github.com/kevva/decompress-unzip/blob/0c33871268e2eee3cb5370bec8ae8a82d887f5d7/index.js#L44).

As a result, you can only decompress files where the file itself _summed with_ its decompressed contents is less than the available memory of your system (`Buffer`s come from available memory, not from V8 heap, so aren't limited by `--max-old-space-size`, but are of course still limited by the amount of available memory the system has).

This is fine on development machines, or very large servers, but even on a `medium` EC2 instance that has 1.5GB of available memory, you're limited to extract files under 1.5GB (this includes their extracted content). On a `nano` instance you can only decompress files under 30MB before you get a 137 exit code (depending on the compression ratio) 😅

_- breathes -_

Do you see a future where `decompress` uses streams instead? (For backward compatibility you could always then buffer everything from the streams -- but at least users would have the option to opt-out of the "buffer everything in memory" behaviour if needed.)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with index.js around the linked input-loading code and decompress-unzip/index.js around the linked extracted-file handling. Trace how buffers move through both packages, then define a streaming or opt-out path that avoids retaining all data while preserving existing behavior and verify it with the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.