aws / aws/aws-encryption-sdk-javascript

decryptStream causes unbounded memory growth

Open Beginner friendly
#1,656 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
260
Forks
68
Avg merge
22h 19m
Merged PRs (30d)
2

Description

### Problem:

While using `decryptStream` to stream-decrypt large S3 objects, I noticed a memory increase roughly equal to the size of the object.

I think the issue is that `_decryptStream` returns a duplexify wrapper and internally runs a pipeline:

```
const stream = new Duplexify(parseHeaderStream, decipherStream)

pipeline(
parseHeaderStream,
verifyStream,
decipherStream,
new PassThrough(),
(err: Error) => {
if (err) stream.emit('error', err)
}
)
```

The caller reads from `decipherStream` via the duplexify wrapper. The pipeline also pushes `decipherStream`'s output into the `PassThrough`. Since nothing ever reads from that `PassThrough`, its internal buffer appears to grow without bound.

### Solution:

Replacing the `PassThrough` with a no-op `Writable` that discards chunks fixes the memory growth while still absorbing the `destroy()` call:

```
const drain = new Writable({
write(_chunk, _encoding, callback) {
callback()
},
})

pipeline(
parseHeaderStream,
verifyStream,
decipherStream,
drain,
(err: Error) => {
if (err) stream.emit('error', err)
}
)
```

I tested this change locally and the memory usage stopped increasing while streaming.

Contributor guide

Open the contributing guide

Research direction

Start at the `_decryptStream` entry point and inspect the pipeline connecting `parseHeaderStream`, `verifyStream`, and `decipherStream` to the `PassThrough`. Replace the unread output sink as described, then validate by streaming a large S3 object and confirming memory no longer grows with the object size.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js, typescript
Domain
backend, cloud, stream-processing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.