aws / aws/aws-sdk-java-v2

S3 Async GetObject `toBytes` uses memory 4x the object size

Aperta
#4,392 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
feature-request p2
Lingua principale
Java
Stelle
2.6k
Fork
1k
Merge medio
2g 9h
PR unite (30g)
51

Descrizione

### Describe the bug

Loading an S3 object into memory using [`AsyncResponseTransformer.toBytes()`](https://github.com/aws/aws-sdk-java-v2/blob/69f7191252c26b351f7fb1c5f031948dac43e4c9/core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java#L204-L206) currently has two related problems:

#### Memory: Excessive peak requirements

Peak memory requirement is at **4x the actual size of the S3 object** - loading a 258MB object into memory takes 1070MB of RAM.

Causes:
* The byte array in `ByteArrayOutputStream` [doubles in size](https://github.com/openjdk/jdk/blob/218829e0a2a3ae5599b81733df53557966392033/src/java.base/share/classes/java/io/ByteArrayOutputStream.java#L100-L101) every time capacity is exhausted - for an object just over 1GB in size, the buffer will grow to 2GB in size
* `ByteArrayOutputStream.toByteArray()` [allocates a new byte array for the result](https://github.com/openjdk/jdk/blob/218829e0a2a3ae5599b81733df53557966392033/src/java.base/share/classes/java/io/ByteArrayOutputStream.java#L186-L188), while the internal byte array is still in place - for an object just over 1GB in size, the internal buffer is 2GB in size, the new result array is 1GB, leading to 3GB being held in memory in simultaneously.
* `ResponseBytes.fromByteArray()` also allocates a new full-size array. The `ByteArrayOutputStream` _may_ have been recovered by this time, but this still requires 2 copies of the object to simultaneously exist - for a 1GB object, 2GB must be held in memory in simultaneously.
* Reading chunks of the response also has some unnecessary additional memory requirements, proportional to the chunk size.

#### CPU: Unnecessary `byte[]` allocations, copies and GC
This problem drives the first problem above, but is also a problem in its own right:

* Copying bytes takes CPU time (eg `System.arraycopy()` for 40MB of bytes takes ~2ms on my M1 machine)
* Garbage collecting the created-&-discarded byte arrays also takes time

### Expected Behavior

Loading a 258MB file with a JVM heap size **less than 2x that size** (as set by `-Xmx`) should consistently succeed.

### Current Behavior

The current AWS SDK implementation requires up to 1070MB to consistently succeed in downloading a 258 MB object from S3 into memory (**4.1x** times the object size). With less JVM heap than that, [the JVM runs out of Java heap space, and crashes](https://github.com/rtyley/aws-sdk-async-response-bytes/actions/runs/6077382438/job/16486966475#step:4:23):

```
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Java heap space`
...
Caused by: java.lang.OutOfMemoryError: Java heap space
```

### Reproduction Steps

Full reproduction is in https://github.com/rtyley/aws-sdk-async-response-bytes, which has a GitHub workflow performing [automated Memory-Consumption tests](https://github.com/rtyley/aws-sdk-async-response-bytes#automated-memory-consumption-tests) - the test script repeatedly downloads a 258 MB object from S3 into memory, while varying the amount of Java heap memory allocated with -Xmx, to find the amount of memory necessary for the download to consistently succeed with the given approach. As an example, [this run with the standard AWS SDK took **1070MB** of heap](https://github.com/rtyley/aws-sdk-async-response-bytes/actions/runs/6058058800#summary-16439830734).

The key code is just [this](https://github.com/rtyley/aws-sdk-async-response-bytes/blob/main/src/main/java/com/madgag/aws/sdk/async/responsebytes/Main.java) - there's nothing special about it:

```
ResponseBytes responseBytes = s3Client.getObject(
GetObjectRequest.builder().bucket(knownS3Object.bucket()).key(knownS3Object.key()).build(),
AsyncResponseTransformer.toBytes()
).get();
```

### Possible Solution

See https://github.com/aws/aws-sdk-java-v2/pull/4355

### Additional Information/Context

_No response_

### AWS Java SDK version used

2.20.120

### JDK version used

corretto 17.0.8.1

### Operating System and version

Ubuntu 22.04.3 LTS

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Read core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java and inspect toBytes(). Compare the behavior with pull request 4355 and run the external reproduction and memory-consumption tests for a 258 MB S3 object. Done means the download consistently succeeds with a JVM heap below twice the object size and avoids the reported excess allocations.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, java
Ambito
api, cloud
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.