Support for zero-copy file upload/download with `java.nio`
- Lingua principale
- Java
- Stelle
- 2.6k
- Fork
- 1k
- Merge medio
- 2g 9h
- PR unite (30g)
- 51
Descrizione
### Describe the feature
Currently it seems that the code specifically deals in on heap ByteBuffers either converting [direct to non-direct Buffers](https://github.com/aws/aws-sdk-java-v2/blob/2.20.49/core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncRequestBody.java#L164) or [writing to non-direct Buffers](https://github.com/aws/aws-sdk-java-v2/blob/2.20.49/core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncRequestBody.java#L258-L259) when there are more performant and memory efficient zero-copy routes.
Most of the heavy lifting on supporting these off heap/zero copy file transfers are done by the HTTP client and the OS with native support in Java via `FileChannel#transferTo` and `FileChannel#transferFrom` to socket channels and similar for [SocketChannel#write](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/SocketChannel.html#write(java.nio.ByteBuffer)) for DirectByteBuffers.
The specific implementation of this is up to the HTTP client in use (e.g. https://netty.io/4.0/api/io/netty/channel/FileRegion.html) but the option is removed at too high a level for the client specific implementations.
I believe it was done this way to guard against concurrent modification during write ops which I've discussed on https://github.com/aws/aws-sdk-java-v2/pull/3925
### Use Case
High frequency efficient read/write ops to S3
### Proposed Solution
Don't do any conversion from direct to non-direct ByteBuffers, pass along the File/FileChannel object instead of converting to a ByteBuffer publisher and leave it up to the http client lib to deal with as required.
For Path/File/Filechannel `AsyncRequestBody` is unsuitable since it's an implementation of `SdkPublisher`, I think in this case the `AsyncRequestBody` is just unnecessary and the plain Path/File/Filechannel should be passed on to the http client.
The Netty docs state
> If your operating system (or JDK / JRE) does not support zero-copy file transfer, sending a file with [FileRegion](https://netty.io/4.0/api/io/netty/channel/FileRegion.html) might fail or yield worse performance. For example, sending a large file doesn't work well in Windows.
In which case this should be possible to disable with a flag passed to the http client when building the S3Client
### Other Information
_No response_
### Acknowledgements
- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### AWS Java SDK version used
macOs 13.2.1 (22D68)
### JDK version used
openjdk 11.0.16.1 & openjdk 19.0.1
### Operating System and version
macOs 13.2.1 (22D68)
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia leggendo AsyncRequestBody.java e FileAsyncRequestBody.java, quindi esamina la discussione nella pull request 3925 sulla modifica concorrente durante le scritture. Traccia il percorso degli input Path/File/FileChannel fino al client HTTP e il modo in cui viene costruita la configurazione di S3Client. Il lavoro è completato quando upload e download con supporto zero-copy possono raggiungere implementazioni specifiche del client, con un'opzione per gli ambienti in cui zero-copy offre prestazioni scarse.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- backend-api-design, cloud
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 25/100