aws / aws/aws-sdk-java-v2

Support for zero-copy file upload/download with `java.nio`

オープン
#3,928 コメント 6 件 リアクション 1 件 担当者 0 名 GitHub で見る
feature-request p3
主要言語
Java
スター
2.6k
フォーク
1k
平均マージ
2日 9時間
マージ済み PR(30日)
51

説明

### Describe the feature

Currently it seems that the code specifically deals in on heap ByteBuffers either converting [direct to non-direct Buffers](https://github.com/aws/aws-sdk-java-v2/blob/2.20.49/core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncRequestBody.java#L164) or [writing to non-direct Buffers](https://github.com/aws/aws-sdk-java-v2/blob/2.20.49/core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncRequestBody.java#L258-L259) when there are more performant and memory efficient zero-copy routes.

Most of the heavy lifting on supporting these off heap/zero copy file transfers are done by the HTTP client and the OS with native support in Java via `FileChannel#transferTo` and `FileChannel#transferFrom` to socket channels and similar for [SocketChannel#write](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/SocketChannel.html#write(java.nio.ByteBuffer)) for DirectByteBuffers.

The specific implementation of this is up to the HTTP client in use (e.g. https://netty.io/4.0/api/io/netty/channel/FileRegion.html) but the option is removed at too high a level for the client specific implementations.

I believe it was done this way to guard against concurrent modification during write ops which I've discussed on https://github.com/aws/aws-sdk-java-v2/pull/3925

### Use Case

High frequency efficient read/write ops to S3

### Proposed Solution

Don't do any conversion from direct to non-direct ByteBuffers, pass along the File/FileChannel object instead of converting to a ByteBuffer publisher and leave it up to the http client lib to deal with as required.

For Path/File/Filechannel `AsyncRequestBody` is unsuitable since it's an implementation of `SdkPublisher`, I think in this case the `AsyncRequestBody` is just unnecessary and the plain Path/File/Filechannel should be passed on to the http client.

The Netty docs state
> If your operating system (or JDK / JRE) does not support zero-copy file transfer, sending a file with [FileRegion](https://netty.io/4.0/api/io/netty/channel/FileRegion.html) might fail or yield worse performance. For example, sending a large file doesn't work well in Windows.

In which case this should be possible to disable with a flag passed to the http client when building the S3Client

### Other Information

_No response_

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### AWS Java SDK version used

macOs 13.2.1 (22D68)

### JDK version used

openjdk 11.0.16.1 & openjdk 19.0.1

### Operating System and version

macOs 13.2.1 (22D68)

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず AsyncRequestBody.java と FileAsyncRequestBody.java を読み、次に書き込み中の同時変更に関する pull request 3925 の議論を確認します。Path/File/FileChannel の入力が HTTP client に到達するまでの流れと、S3Client の設定がどのように構築されるかを追跡します。zero-copy に対応可能なアップロードとダウンロードがクライアント固有の実装に到達でき、zero-copy の性能が低い環境向けのオプションが存在すれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
backend-api-design, cloud
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。