aws / aws/aws-sdk-java-v2

UploadPartResponse seems to hold a reference to the content of the part, even after stream is closed

Open
#6,273 6 comments 1 reaction 1 assignee Claimed by @debora-ito View on GitHub
bug
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the bug

When using the `S3Client` to `uploadPart`, the returned `UploadPartResponse` seems to hold a reference to the content of the part, causing memory build up over time

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

The `UploadPartResponse` should only hold information related to the actual response as described here: https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html#API_UploadPart_ResponseElements

### Current Behavior

When uploading multiple parts in sequence and keeping the `UploadPartResponse` in memory to be used later to complete the multi part upload we see a build up of memory over time. Upon inspection of a heap dump we noticed that the `UploadPartResponse` had a memory footprint roughly equal to our part size, which was unexpected.

Image

It seems that `UploadPartResponse` holds a reference to an underlying `sdkHttpResponse` which in turn holds a `content` reference which is an `AbortableInputStream` the size of the part.

### Reproduction Steps

Our code is written in Scala, but to the best of my knowledge this Java code serves as a minimum example
```java
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;
import software.amazon.awssdk.core.sync.RequestBody;

import java.io.ByteArrayInputStream;
import java.util.ArrayList;
import java.util.List;

public class S3MultipartUploadFullExample {
public static void main(String[] args) {
S3Client s3 = S3Client.create();
String bucket = "your-bucket";
String key = "your-key";
byte[] chunk = new byte[50 * 1024 * 1024]; // 50MB chunk

CreateMultipartUploadRequest createRequest = CreateMultipartUploadRequest.builder()
.bucket(bucket)
.key(key)
.build();
CreateMultipartUploadResponse initiateResult = s3.createMultipartUpload(createRequest);

// We need to hold the responses so we can complete the multi part upload later
List partUploadResponses = new ArrayList<>();
for (int i = 0; i < 100; i++) {
try (ByteArrayInputStream is = new ByteArrayInputStream(chunk)) {
int partNumber = i + 1;
UploadPartRequest uploadPartRequest = UploadPartRequest.builder()
.bucket(bucket)
.key(key)
.uploadId(initiateResult.uploadId())
.partNumber(partNumber)
.contentLength(chunk.length)
.build();
UploadPartResponse response = s3.uploadPart(uploadPartRequest, RequestBody.fromInputStream(is, chunk.length));
partUploadResponses.add(response);
}
}
// At this point the heap has grown to 100 * 50MB ~ 5GB because each UploadPartResponse
// in partUploadResponses holds a reference to a stream with the content

List completedParts = new ArrayList<>();
for (int i = 0; i < partUploadResponses.size(); i++) {
completedParts.add(CompletedPart.builder()
.partNumber(i + 1)
.eTag(partUploadResponses.get(i).eTag())
.build());
}

CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder()
.parts(completedParts)
.build();
CompleteMultipartUploadRequest completeRequest = CompleteMultipartUploadRequest.builder()
.bucket(bucket)
.key(key)
.uploadId(initiateResult.uploadId())
.multipartUpload(completedMultipartUpload)
.build();
CompleteMultipartUploadResponse completeResponse = s3.completeMultipartUpload(completeRequest);

System.out.println("Multipart upload completed: " + completeResponse.key());
s3.close();
}
}
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS Java SDK version used

2.31.78

### JDK version used

Picked up JAVA_TOOL_OPTIONS: -XX:UseSVE=0 openjdk 21.0.7 2025-04-15 OpenJDK Runtime Environment (build 21.0.7+6-alpine-r0) OpenJDK 64-Bit Server VM (build 21.0.7+6-alpine-r0, mixed mode, sharing)

### Operating System and version

Linux Alpine 3.21.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.