aws / aws/aws-encryption-sdk-javascript
decryptStream causes unbounded memory growth
- Langage dominant
- TypeScript
- Étoiles
- 260
- Forks
- 68
- Merge moyen
- 22 h 19 min
- PR mergées (30 j)
- 2
Description
### Problem:
While using `decryptStream` to stream-decrypt large S3 objects, I noticed a memory increase roughly equal to the size of the object.
I think the issue is that `_decryptStream` returns a duplexify wrapper and internally runs a pipeline:
```
const stream = new Duplexify(parseHeaderStream, decipherStream)
pipeline(
parseHeaderStream,
verifyStream,
decipherStream,
new PassThrough(),
(err: Error) => {
if (err) stream.emit('error', err)
}
)
```
The caller reads from `decipherStream` via the duplexify wrapper. The pipeline also pushes `decipherStream`'s output into the `PassThrough`. Since nothing ever reads from that `PassThrough`, its internal buffer appears to grow without bound.
### Solution:
Replacing the `PassThrough` with a no-op `Writable` that discards chunks fixes the memory growth while still absorbing the `destroy()` call:
```
const drain = new Writable({
write(_chunk, _encoding, callback) {
callback()
},
})
pipeline(
parseHeaderStream,
verifyStream,
decipherStream,
drain,
(err: Error) => {
if (err) stream.emit('error', err)
}
)
```
I tested this change locally and the memory usage stopped increasing while streaming.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez au point d’entrée `_decryptStream` et examinez le pipeline qui relie `parseHeaderStream`, `verifyStream` et `decipherStream` à `PassThrough`. Remplacez le puits de sortie non lu comme indiqué, puis validez en diffusant un objet S3 volumineux et en confirmant que la mémoire n’augmente plus avec la taille de l’objet.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- aws, node.js, typescript
- Domaine
- backend, cloud, stream-processing
- Type d'issue
- Bug
- Difficulté
- 2/5
- Temps estimé
- 1-3 heures
- Activité
- Calme
- Clarté
- Clairement spécifiée
- Accessibilité débutants
- 68/100