Humongous allocation with CRT client when downloading chunks of files
- Lingua principale
- Java
- Stelle
- 2.6k
- Fork
- 1k
- Merge medio
- 2g 9h
- PR unite (30g)
- 51
Descrizione
### Describe the bug
When downloading a 4mb chunk from a large file, we were seeing humongous allocation from native code. CRT client was passing heap bytes `byte[] bodyBytesIn` to the caller.
Can we make it offheap?
Version: aws-crt:0.33.9
Lowering the part size helps to allocate smaller bytes, but still heap pressure.
We are using 8mb as part size. Will lowering it to 512kb affecting partial download or full file downloading performance?
```
class S3MetaRequestResponseHandlerNativeAdapter {
int onResponseBody(byte[] bodyBytesIn, long objectRangeStart, long objectRangeEnd) {
return this.responseHandler.onResponseBody(ByteBuffer.wrap(bodyBytesIn), objectRangeStart, objectRangeEnd);
}
````
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
Offheap buffer should be used
### Current Behavior
Heap buffer is used
### Reproduction Steps
Download 4mb chunk from a large file.
```
GetObjectRequest request = GetObjectRequest.builder()
.bucket(bucket)
.key(rkey)
.range(String.format("bytes=%d-%d", pos, pos+size-1))
.build();
return s3Async.getObject(request, asyncResponseTransformerImp);
```
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### AWS Java SDK version used
aws-crt:0.33.9
### JDK version used
JDK22
### Operating System and version
Ubuntu 22.04 LTS. Also reproduced on Mac M1.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.