aws / aws/amazon-s3-encryption-client-java

OOM and request cancellation when reading large objects

Aperta
#484 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub
good first issue
Lingua principale
Java
Stelle
34
Fork
21
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### Problem:

When attempting to read a large file (~1GB) from S3 using the v3 Encryption SDK, the request fails with a hidden OutOfMemoryError. In v2 SDK, the same operation works without issues.

Stack trace:
```
java.io.IOException: Request cancelled
at software.amazon.awssdk.http.nio.netty.internal.FutureCancelHandler.exceptionCaught(FutureCancelHandler.java:63)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1221)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1324)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:856)
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$createExecutionFuture$3(NettyRequestExecutor.java:135)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:840)
```
The exception reported is misleading (FutureCancelledException), but after debugging I found it’s masking an actual OOM:
```
software.amazon.awssdk.http.nio.netty.internal.FutureCancelledException: java.lang.OutOfMemoryError: Java heap space
```

Steps to reproduce:
```java
S3EncryptionClient s3EncryptionClient = S3EncryptionClient.builder()
.enableDelayedAuthenticationMode(true)
.kmsKeyId("XXX")
.build();
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
.bucket("XXX")
.key("XXX")
.build();
try (BufferedReader stream = new BufferedReader(new InputStreamReader(
s3EncryptionClient.getObject(getObjectRequest, ResponseTransformer.toInputStream())))) {
String line;
while ((line = stream.readLine()) != null) {
System.out.println("Read line: " + line);
// Process the line
}
}
```

I am using the latest version 3.4.0. The file size being tested is approximately 1GB. The runtime environment is Java 17.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with the S3EncryptionClient.getObject call using ResponseTransformer.toInputStream and reproduce the 1GB read on Java 17 with delayed authentication enabled. Trace where the hidden OutOfMemoryError becomes FutureCancelledException; done means large-object reads no longer exhaust the heap and the reported failure is accurate, with a regression test for the reproduction.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, java
Ambito
backend, security
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.