aws / aws/amazon-s3-encryption-client-java
Multipart upload with CSE using RSA key pair
- Dominant language
- Java
- Stars
- 34
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
### Problem:
My code is roughly
```
val s3ClientObject =
S3Client
.builder()
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(
spec.accessKey.get,
getSecretKey(metadataEncryptionUtils)
)
)
)
.region(REGION.US_EAST_1).build()
val s3AsyncClientObject =
S3AsyncClient
.builder()
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(
spec.accessKey.get,
getSecretKey(metadataEncryptionUtils)
)
)
)
.region(REGION.US_EAST_1).build()
```
Now I create S3EncryptionClient by wrapping above such as
```
val encObject =
S3EncryptionClient
.builder()
.rsaKeyPair(userKeys)
.enableLegacyUnauthenticatedModes(true)
.enableLegacyWrappingAlgorithms(true)
.wrappedClient(s3ClientObject)
.wrappedAsyncClient(s3AsyncClientObject)
.enableDelayedAuthenticationMode(true)
.build()
```
I am able to use this `encObject` to do operations like creating bucket etc. I am also able to upload files to s3 bucket.
However, when i try to upload a large file ( say ~200MB ) with multi part upload it fails with following error
```
aused by: software.amazon.awssdk.crt.http.HttpException: Amount of data streamed out does not match the previously declared length.
at software.amazon.awssdk.http.crt.internal.response.CrtResponseAdapter.onResponseComplete(CrtResponseAdapter.java:108) ~[thirdparty-intellij-deps.jar:?]
at software.amazon.awssdk.crt.http.HttpStreamResponseHandlerNativeAdapter.onResponseComplete(HttpStreamResponseHandlerNativeAdapter.java:58) ~[thirdparty-intellij-deps.jar:?]
Exception in thread "AwsEventLoop 9" java.lang.IllegalStateException: Encountered fatal error in publisher
at software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
at software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:76)
at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:108)
at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:81)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Must use either different key or iv for GCM encryption
at java.base/com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
at java.base/com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
at java.base/com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at java.base/javax.crypto.Cipher.update(Cipher.java:1869)
at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
... 10 more
```
If I directly use the `s3ClientObject` it works.
### Solution:
Is there some limitation on CSE with multi part uploads ?
Contributor guide
Research direction
Reproduce the roughly 200 MB multipart upload using the S3EncryptionClient builder with rsaKeyPair, then compare it with the direct s3ClientObject path. Start from the CipherSubscriber stack trace and multipart upload flow; done means encrypted multipart uploads complete without the declared-length or GCM key/IV errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend-api-design, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100