S3 Async GetObject `toBytes` uses memory 4x the object size
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 51
Description
### Describe the bug
Loading an S3 object into memory using [`AsyncResponseTransformer.toBytes()`](https://github.com/aws/aws-sdk-java-v2/blob/69f7191252c26b351f7fb1c5f031948dac43e4c9/core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java#L204-L206) currently has two related problems:
#### Memory: Excessive peak requirements
Peak memory requirement is at **4x the actual size of the S3 object** - loading a 258MB object into memory takes 1070MB of RAM.
Causes:
* The byte array in `ByteArrayOutputStream` [doubles in size](https://github.com/openjdk/jdk/blob/218829e0a2a3ae5599b81733df53557966392033/src/java.base/share/classes/java/io/ByteArrayOutputStream.java#L100-L101) every time capacity is exhausted - for an object just over 1GB in size, the buffer will grow to 2GB in size
* `ByteArrayOutputStream.toByteArray()` [allocates a new byte array for the result](https://github.com/openjdk/jdk/blob/218829e0a2a3ae5599b81733df53557966392033/src/java.base/share/classes/java/io/ByteArrayOutputStream.java#L186-L188), while the internal byte array is still in place - for an object just over 1GB in size, the internal buffer is 2GB in size, the new result array is 1GB, leading to 3GB being held in memory in simultaneously.
* `ResponseBytes.fromByteArray()` also allocates a new full-size array. The `ByteArrayOutputStream` _may_ have been recovered by this time, but this still requires 2 copies of the object to simultaneously exist - for a 1GB object, 2GB must be held in memory in simultaneously.
* Reading chunks of the response also has some unnecessary additional memory requirements, proportional to the chunk size.
#### CPU: Unnecessary `byte[]` allocations, copies and GC
This problem drives the first problem above, but is also a problem in its own right:
* Copying bytes takes CPU time (eg `System.arraycopy()` for 40MB of bytes takes ~2ms on my M1 machine)
* Garbage collecting the created-&-discarded byte arrays also takes time
### Expected Behavior
Loading a 258MB file with a JVM heap size **less than 2x that size** (as set by `-Xmx`) should consistently succeed.
### Current Behavior
The current AWS SDK implementation requires up to 1070MB to consistently succeed in downloading a 258 MB object from S3 into memory (**4.1x** times the object size). With less JVM heap than that, [the JVM runs out of Java heap space, and crashes](https://github.com/rtyley/aws-sdk-async-response-bytes/actions/runs/6077382438/job/16486966475#step:4:23):
```
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Java heap space`
...
Caused by: java.lang.OutOfMemoryError: Java heap space
```
### Reproduction Steps
Full reproduction is in https://github.com/rtyley/aws-sdk-async-response-bytes, which has a GitHub workflow performing [automated Memory-Consumption tests](https://github.com/rtyley/aws-sdk-async-response-bytes#automated-memory-consumption-tests) - the test script repeatedly downloads a 258 MB object from S3 into memory, while varying the amount of Java heap memory allocated with -Xmx, to find the amount of memory necessary for the download to consistently succeed with the given approach. As an example, [this run with the standard AWS SDK took **1070MB** of heap](https://github.com/rtyley/aws-sdk-async-response-bytes/actions/runs/6058058800#summary-16439830734).
The key code is just [this](https://github.com/rtyley/aws-sdk-async-response-bytes/blob/main/src/main/java/com/madgag/aws/sdk/async/responsebytes/Main.java) - there's nothing special about it:
```
ResponseBytes responseBytes = s3Client.getObject(
GetObjectRequest.builder().bucket(knownS3Object.bucket()).key(knownS3Object.key()).build(),
AsyncResponseTransformer.toBytes()
).get();
```
### Possible Solution
See https://github.com/aws/aws-sdk-java-v2/pull/4355
### Additional Information/Context
_No response_
### AWS Java SDK version used
2.20.120
### JDK version used
corretto 17.0.8.1
### Operating System and version
Ubuntu 22.04.3 LTS
Contributor guide
Assessment
This issue has not been assessed yet.