aws / aws/amazon-s3-encryption-client-java

OOM and request cancellation when reading large objects

Open
#484 6 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Java
Stars
34
Forks
21
PR merge metrics
No merged PRs in 30d

Description

### Problem:

When attempting to read a large file (~1GB) from S3 using the v3 Encryption SDK, the request fails with a hidden OutOfMemoryError. In v2 SDK, the same operation works without issues.

Stack trace:
```
java.io.IOException: Request cancelled
at software.amazon.awssdk.http.nio.netty.internal.FutureCancelHandler.exceptionCaught(FutureCancelHandler.java:63)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1221)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1324)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:856)
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$createExecutionFuture$3(NettyRequestExecutor.java:135)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:840)
```
The exception reported is misleading (FutureCancelledException), but after debugging I found it’s masking an actual OOM:
```
software.amazon.awssdk.http.nio.netty.internal.FutureCancelledException: java.lang.OutOfMemoryError: Java heap space
```

Steps to reproduce:
```java
S3EncryptionClient s3EncryptionClient = S3EncryptionClient.builder()
.enableDelayedAuthenticationMode(true)
.kmsKeyId("XXX")
.build();
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
.bucket("XXX")
.key("XXX")
.build();
try (BufferedReader stream = new BufferedReader(new InputStreamReader(
s3EncryptionClient.getObject(getObjectRequest, ResponseTransformer.toInputStream())))) {
String line;
while ((line = stream.readLine()) != null) {
System.out.println("Read line: " + line);
// Process the line
}
}
```

I am using the latest version 3.4.0. The file size being tested is approximately 1GB. The runtime environment is Java 17.

Contributor guide

Open the contributing guide

Research direction

Start with the S3EncryptionClient.getObject call using ResponseTransformer.toInputStream and reproduce the 1GB read on Java 17 with delayed authentication enabled. Trace where the hidden OutOfMemoryError becomes FutureCancelledException; done means large-object reads no longer exhaust the heap and the reported failure is accurate, with a regression test for the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.