aws / aws/aws-sdk-java-v2

Streaming of S3 large files using S3AsyncClient and AsyncResponseTransformer.toPublisher(), CRT vs netty backpressure

Open
#5,158 14 comments 0 reactions 0 assignees View on GitHub
documentation p2
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the bug

To stream data directly instead of writing it to a file or into a memory #391 introduced `AsyncResponseTransformer.toPublisher()` method which essentially provides a way to return `Mono>` to the caller.

The javadoc however, clearly mentions that: "You are responsible for subscribing to this publisher and managing the associated back-pressure. Therefore, this transformer is only recommended for advanced use cases."

I am attaching a project which has two things: Spring Boot server which can download large file from S3 using S3AsyncClient (use `spring.profiles.active` in `application.yml` to use either CRT or Netty client) and very slow consumer.
[s3-streaming.zip](https://github.com/aws/aws-sdk-java-v2/files/15134143/s3-streaming.zip)

If you specify a bucket and large file in the application.yml, start the application with CRT client (default) and start slow client you may observe that in the log, every 10 seconds spring reactor netty direct memory consumption will be printed, it will quickly fill with the size of the file while consumer slowly keeps reading it.

Now, if you have 10 slow consumers, your direct mem will quickly hit the wall and requests will start to error out, causing Premature End of Content and closed connections.

I was not able to find any examples on how to maintain backpressure as `Flux.limitRate` didn't work at all.

CRT client was chosen by us based on the [AWS http configuration page](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration.html).

It would be nice if someone can look at this to figure out if there is an issue with the code itself, or code itself is fine and this use case doesn't play well with CRT.

I need to also mention that once CRT is replaced with netty, memory consumption stays minimal, so backpressure seems to work without any manual intervention if netty is used on both sides (reactor-netty and s3 sdk netty). However, we observed a different issue with that - under load, some of the requests are idling and closed by a timeout on the ALB side, it is under investigation.

The point is, while documentation says about backpressure, it seems to work if using netty as s3 http client and doesn't work if using CRT as client. Please advice.

### Expected Behavior

CRT s3 http client behaves similar to s3 netty http client and doesn't push all the data to server direct buffers.

### Current Behavior

CRT instantly pushes all data to the server's netty direct byte buffers for slow consumers.
That doesn't happen if using netty http client instead of CRT.

### Reproduction Steps

Use the attached zip file maven project.
Update application.yml to point to s3 bucket and file which is larger than 200MB.
You may also need to specify AWS credentials provider in the code or in the system.

Start the server. Start the SlowDrainApplication main method.
Observe the server logs to see Netty: direct mem consumption.
All data will get pushed into the direct memory.

Go to application.yml and change spring.profiles.active from 'crt' to 'netty'.
Repeat the exercise.
Observe the server logs to see that no excess memory consumption is happening without any additional backpressure configuration.

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS Java SDK version used

2.25.10

### CRT Client Version

0.29.14

### Reactor netty and netty version
reactor-netty 1.1.3
netty 4.1.108.Final

### JDK version used

openjdk version "17.0.3" 2022-04-19
oracle jdk 17

### Operating System and version

windows 11 23H2 and some linux image in EKS

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.