aws / aws/aws-sdk-java-v2

Humongous allocation with CRT client when downloading chunks of files

Aperta
#6,566 7 commenti 0 reazioni 1 assegnatario Rivendicata da @debora-ito Vedi su GitHub
bug crt-client
Lingua principale
Java
Stelle
2.6k
Fork
1k
Merge medio
2g 9h
PR unite (30g)
51

Descrizione

### Describe the bug

When downloading a 4mb chunk from a large file, we were seeing humongous allocation from native code. CRT client was passing heap bytes `byte[] bodyBytesIn` to the caller.

Can we make it offheap?

Version: aws-crt:0.33.9

Lowering the part size helps to allocate smaller bytes, but still heap pressure.

We are using 8mb as part size. Will lowering it to 512kb affecting partial download or full file downloading performance?

```
class S3MetaRequestResponseHandlerNativeAdapter {

int onResponseBody(byte[] bodyBytesIn, long objectRangeStart, long objectRangeEnd) {
return this.responseHandler.onResponseBody(ByteBuffer.wrap(bodyBytesIn), objectRangeStart, objectRangeEnd);
}

````

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

Offheap buffer should be used

### Current Behavior

Heap buffer is used

### Reproduction Steps

Download 4mb chunk from a large file.

```
GetObjectRequest request = GetObjectRequest.builder()
.bucket(bucket)
.key(rkey)
.range(String.format("bytes=%d-%d", pos, pos+size-1))
.build();

return s3Async.getObject(request, asyncResponseTransformerImp);
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS Java SDK version used

aws-crt:0.33.9

### JDK version used

JDK22

### Operating System and version

Ubuntu 22.04 LTS. Also reproduced on Mac M1.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.