aws / aws/aws-sdk-java-v2

Humongous allocation with CRT client when downloading chunks of files

Open
#6,566 7 comments 0 reactions 1 assignee Claimed by @debora-ito View on GitHub
bug crt-client
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the bug

When downloading a 4mb chunk from a large file, we were seeing humongous allocation from native code. CRT client was passing heap bytes `byte[] bodyBytesIn` to the caller.

Can we make it offheap?

Version: aws-crt:0.33.9

Lowering the part size helps to allocate smaller bytes, but still heap pressure.

We are using 8mb as part size. Will lowering it to 512kb affecting partial download or full file downloading performance?

```
class S3MetaRequestResponseHandlerNativeAdapter {

int onResponseBody(byte[] bodyBytesIn, long objectRangeStart, long objectRangeEnd) {
return this.responseHandler.onResponseBody(ByteBuffer.wrap(bodyBytesIn), objectRangeStart, objectRangeEnd);
}

````

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

Offheap buffer should be used

### Current Behavior

Heap buffer is used

### Reproduction Steps

Download 4mb chunk from a large file.

```
GetObjectRequest request = GetObjectRequest.builder()
.bucket(bucket)
.key(rkey)
.range(String.format("bytes=%d-%d", pos, pos+size-1))
.build();

return s3Async.getObject(request, asyncResponseTransformerImp);
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS Java SDK version used

aws-crt:0.33.9

### JDK version used

JDK22

### Operating System and version

Ubuntu 22.04 LTS. Also reproduced on Mac M1.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.