aws / aws/amazon-s3-encryption-client-java
Multipart upload with CSE using RSA key pair
- 主要语言
- Java
- 星标
- 34
- 派生
- 21
- PR 合并指标
- 30 天内没有已合并 PR
描述
### Problem:
My code is roughly
```
val s3ClientObject =
S3Client
.builder()
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(
spec.accessKey.get,
getSecretKey(metadataEncryptionUtils)
)
)
)
.region(REGION.US_EAST_1).build()
val s3AsyncClientObject =
S3AsyncClient
.builder()
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(
spec.accessKey.get,
getSecretKey(metadataEncryptionUtils)
)
)
)
.region(REGION.US_EAST_1).build()
```
Now I create S3EncryptionClient by wrapping above such as
```
val encObject =
S3EncryptionClient
.builder()
.rsaKeyPair(userKeys)
.enableLegacyUnauthenticatedModes(true)
.enableLegacyWrappingAlgorithms(true)
.wrappedClient(s3ClientObject)
.wrappedAsyncClient(s3AsyncClientObject)
.enableDelayedAuthenticationMode(true)
.build()
```
I am able to use this `encObject` to do operations like creating bucket etc. I am also able to upload files to s3 bucket.
However, when i try to upload a large file ( say ~200MB ) with multi part upload it fails with following error
```
aused by: software.amazon.awssdk.crt.http.HttpException: Amount of data streamed out does not match the previously declared length.
at software.amazon.awssdk.http.crt.internal.response.CrtResponseAdapter.onResponseComplete(CrtResponseAdapter.java:108) ~[thirdparty-intellij-deps.jar:?]
at software.amazon.awssdk.crt.http.HttpStreamResponseHandlerNativeAdapter.onResponseComplete(HttpStreamResponseHandlerNativeAdapter.java:58) ~[thirdparty-intellij-deps.jar:?]
Exception in thread "AwsEventLoop 9" java.lang.IllegalStateException: Encountered fatal error in publisher
at software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
at software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:76)
at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:108)
at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:81)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Must use either different key or iv for GCM encryption
at java.base/com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
at java.base/com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
at java.base/com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at java.base/javax.crypto.Cipher.update(Cipher.java:1869)
at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
... 10 more
```
If I directly use the `s3ClientObject` it works.
### Solution:
Is there some limitation on CSE with multi part uploads ?
贡献指南
调研方向
使用带有 rsaKeyPair 的 S3EncryptionClient builder 重现大约 200 MB 的 multipart upload,然后将其与直接的 s3ClientObject 路径进行比较。从 CipherSubscriber 的 stack trace 和 multipart upload 流程入手;完成标准是加密的 multipart upload 能够完成,且不出现 declared-length 或 GCM key/IV 错误。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- aws, java
- 领域
- backend-api-design, security
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100