aws / aws/aws-sdk-java-v2

Support for zero-copy file upload/download with `java.nio`

Offen
#3,928 6 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
feature-request p3
Vorherrschende Sprache
Java
Sterne
2.6k
Forks
1k
Ø Merge
2 T. 9 Std.
Gemergte PRs (30 T.)
51

Beschreibung

### Describe the feature

Currently it seems that the code specifically deals in on heap ByteBuffers either converting [direct to non-direct Buffers](https://github.com/aws/aws-sdk-java-v2/blob/2.20.49/core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncRequestBody.java#L164) or [writing to non-direct Buffers](https://github.com/aws/aws-sdk-java-v2/blob/2.20.49/core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncRequestBody.java#L258-L259) when there are more performant and memory efficient zero-copy routes.

Most of the heavy lifting on supporting these off heap/zero copy file transfers are done by the HTTP client and the OS with native support in Java via `FileChannel#transferTo` and `FileChannel#transferFrom` to socket channels and similar for [SocketChannel#write](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/SocketChannel.html#write(java.nio.ByteBuffer)) for DirectByteBuffers.

The specific implementation of this is up to the HTTP client in use (e.g. https://netty.io/4.0/api/io/netty/channel/FileRegion.html) but the option is removed at too high a level for the client specific implementations.

I believe it was done this way to guard against concurrent modification during write ops which I've discussed on https://github.com/aws/aws-sdk-java-v2/pull/3925

### Use Case

High frequency efficient read/write ops to S3

### Proposed Solution

Don't do any conversion from direct to non-direct ByteBuffers, pass along the File/FileChannel object instead of converting to a ByteBuffer publisher and leave it up to the http client lib to deal with as required.

For Path/File/Filechannel `AsyncRequestBody` is unsuitable since it's an implementation of `SdkPublisher`, I think in this case the `AsyncRequestBody` is just unnecessary and the plain Path/File/Filechannel should be passed on to the http client.

The Netty docs state
> If your operating system (or JDK / JRE) does not support zero-copy file transfer, sending a file with [FileRegion](https://netty.io/4.0/api/io/netty/channel/FileRegion.html) might fail or yield worse performance. For example, sending a large file doesn't work well in Windows.

In which case this should be possible to disable with a flag passed to the http client when building the S3Client

### Other Information

_No response_

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### AWS Java SDK version used

macOs 13.2.1 (22D68)

### JDK version used

openjdk 11.0.16.1 & openjdk 19.0.1

### Operating System and version

macOs 13.2.1 (22D68)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie mit dem Lesen von AsyncRequestBody.java und FileAsyncRequestBody.java und sehen Sie sich anschließend die Diskussion in pull request 3925 über gleichzeitige Änderungen während Schreibvorgängen an. Verfolgen Sie, wie Path/File/FileChannel-Eingaben den HTTP client erreichen und wie die Konfiguration von S3Client erstellt wird. Als abgeschlossen gilt die Arbeit, wenn Uploads und Downloads mit zero-copy-Unterstützung client-spezifische Implementierungen erreichen können und es eine Option für Umgebungen gibt, in denen zero-copy schlecht funktioniert.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
backend-api-design, cloud
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.