aws / aws/aws-sdk-java-v2

Reads on S3 are slower compared to V1 SDK on V2 SDK

Open
#4,807 4 comments 1 reaction 0 assignees View on GitHub
bug p2
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the bug

We can see that the reads on the new SDK are slower than the v1 sdk. On profiling a long running job it was revealed that new SDK has checksum computation on the InputStream which leads to a slower read performance. The ChecksumValidatingInputStream is initialized by default and the client config overrides don't work for it.
```
private static ExecutionAttributes mergeExecutionAttributeOverrides(
ExecutionAttributes executionAttributes,
ExecutionAttributes clientOverrideExecutionAttributes,
ExecutionAttributes requestOverrideExecutionAttributes) {

executionAttributes.putAbsentAttributes(requestOverrideExecutionAttributes);
executionAttributes.putAbsentAttributes(clientOverrideExecutionAttributes);

return executionAttributes;
}
```
**The checksum computation takes around ~10% percent of the time of the overall job, meaning that the reads will in itself be much slower in actuality.**
e.g.
with new sdk:
```
Download Summary
================
Data size 10,737,418,240 bytes
Download duration 0:03:40.848
```
with older sdk:
```
Download Summary
================
Data size 10,737,418,240 bytes
Download duration 0:02:52.134
```

Attaching the profiling screenshots of a job with bandwidth command run at a scale of 10G, they are html flame graphs and can be opened using chrome.
[flamegraph_v1_sdk.txt](https://github.com/aws/aws-sdk-java-v2/files/13808827/flamegraph_v1_sdk.txt)
[flamegraph_v2_sdk.txt](https://github.com/aws/aws-sdk-java-v2/files/13808832/flamegraph_v2_sdk.txt)

### Expected Behavior

1. We should be **able to turn off the ChecksumValidatingInputStream** for reads
2. We should be **aware of such regressions on the documentation itself when the core behaviour is changed** on the service side
3. We should not see **slow downs** even if a new feature is added to the existing functions.

### Current Behavior

Mentioned Above

### Reproduction Steps

Get a distribution of hadoop from https://hadoop.apache.org/releases.html compatible with v2 sdk
Get an older distribution of hadoop from: https://hadoop.apache.org/release/3.3.4.html
Get the latest release of cloudstore: https://github.com/steveloughran/cloudstore/releases
Configure the core-site.xml with your S3 access keys
Run the bandwidth command with the hadoop jar option from both the hadoop versions.

### Possible Solution

Mentioned Above.

### Additional Information/Context

_No response_

### AWS Java SDK version used

2.21.46

### JDK version used

1.8

### Operating System and version

CentOS7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.