aws / aws/aws-sdk-java-v2

Unable to use 2g files as the content for an UploadPartRequest when performing multipart uploads

Open
#6,666 1 comment 0 reactions 0 assignees View on GitHub
bug crt-client p2
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the bug

Unable to perform very large multipart uploads as the largest size for an individual part is `Integer.MAX_VALUE`.

The docs say the current limits for a part should be [5g](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) and I could uploads parts of at least 2g via the CLI - just not via the java SDK

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

The multipart upload to work for large parts

### Current Behavior

Fails with the error

```
Exception in thread "AwsEventLoop 6" java.lang.NullPointerException: Cannot invoke "java.nio.ByteBuffer.hasRemaining()" because "out" is null
at software.amazon.awssdk.utils.async.ByteBufferStoringSubscriber.transferTo(ByteBufferStoringSubscriber.java:100)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtRequestBodyStreamAdapter.sendRequestBody(S3CrtRequestBodyStreamAdapter.java:47)
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure. (SDK Attempt Count: 1)
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure. (SDK Attempt Count: 1)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
... 25 more
```

When debug was enabled on the CRT client the logs had an entry with error code 2064 but little else

### Reproduction Steps

Create 2 files via
```
mkfile -n 2147483647 file2g_1
mkfile -n 2g file2g
```

Then pass these paths (and their file size of `(long) Integer.MAX_VALUE` and `Integer.MAX_VALUE + 1L`) to the following bit of code
```
private void reproducer(String bucket, String s3Key, Path filePath, long fileSize) throws Exception {
try (var client = S3AsyncClient.crtBuilder().build()) {
var uploadId = client.createMultipartUpload(CreateMultipartUploadRequest.builder()
.bucket(bucket)
.key(s3Key)
.build())
.get()
.uploadId();

var partRequest = UploadPartRequest.builder()
.bucket(bucket)
.key(s3Key)
.uploadId(uploadId)
.partNumber(1)
.contentLength(fileSize)
.build();

var eTag = client.uploadPart(partRequest, fromFile(filePath)).get().eTag();
var parts = List.of(CompletedPart.builder()
.partNumber(1)
.eTag(eTag)
.build());

client.completeMultipartUpload(CompleteMultipartUploadRequest.builder()
.bucket(bucket)
.key(s3Key)
.uploadId(uploadId)
.multipartUpload(CompletedMultipartUpload.builder()
.parts(parts)
.build())
.build())
.get();
}
}
```

It works OK for `file2g_1` but fails on `file2g`.

### Possible Solution

_No response_

### Additional Information/Context

I tried via the input stream method of `AsyncRequestBody` and that failed too

I had a suspicion it might be related to the checksum calculations following some step-debugging but changing the algorithms had no effect sadly

I also tried with a file of size 4g to check it wasn't a boundary issue on Integer.MAX but that failed as well.

### AWS Java SDK version used

2.41.4

### JDK version used

21

### Operating System and version

Mac and Linux (EC2)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.