aws / aws/aws-encryption-sdk-javascript
decryptStream causes unbounded memory growth
- Dominant language
- TypeScript
- Stars
- 260
- Forks
- 68
- Avg merge
- 22h 19m
- Merged PRs (30d)
- 2
Description
### Problem:
While using `decryptStream` to stream-decrypt large S3 objects, I noticed a memory increase roughly equal to the size of the object.
I think the issue is that `_decryptStream` returns a duplexify wrapper and internally runs a pipeline:
```
const stream = new Duplexify(parseHeaderStream, decipherStream)
pipeline(
parseHeaderStream,
verifyStream,
decipherStream,
new PassThrough(),
(err: Error) => {
if (err) stream.emit('error', err)
}
)
```
The caller reads from `decipherStream` via the duplexify wrapper. The pipeline also pushes `decipherStream`'s output into the `PassThrough`. Since nothing ever reads from that `PassThrough`, its internal buffer appears to grow without bound.
### Solution:
Replacing the `PassThrough` with a no-op `Writable` that discards chunks fixes the memory growth while still absorbing the `destroy()` call:
```
const drain = new Writable({
write(_chunk, _encoding, callback) {
callback()
},
})
pipeline(
parseHeaderStream,
verifyStream,
decipherStream,
drain,
(err: Error) => {
if (err) stream.emit('error', err)
}
)
```
I tested this change locally and the memory usage stopped increasing while streaming.
Contributor guide
Research direction
Start at the `_decryptStream` entry point and inspect the pipeline connecting `parseHeaderStream`, `verifyStream`, and `decipherStream` to the `PassThrough`. Replace the unread output sink as described, then validate by streaming a large S3 object and confirming memory no longer grows with the object size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, node.js, typescript
- Domain
- backend, cloud, stream-processing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100