aws / aws/amazon-s3-encryption-client-java
S3EncryptionClientException: Re-subscription is not supported during Multipart Upload retry (InputStream source)
- Dominant language
- Java
- Stars
- 34
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
### Problem:
We are encountering a critical failure when performing a high-level Multipart Upload (putObject) using the AWS S3 Encryption Client. The operation fails specifically during SDK automatic retry attempts triggered by network latency.
Stack trace of failure:
```
software.amazon.encryption.s3.S3EncryptionClientException: Exception while performing Multipart Upload PutObject
at software.amazon.encryption.s3.S3EncryptionClient.putObject(S3EncryptionClient.java:198) ~[sd-0.1.jar:?]
at ...
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) ~[?:?]
at scala.util.Success.$anonfun$map$1(Try.scala:255) ~[?:?]
at scala.util.Success.map(Try.scala:213) ~[?:?]
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) ~[?:?]
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) ~[?:?]
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) ~[?:?]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: software.amazon.encryption.s3.S3EncryptionClientException: software.amazon.encryption.s3.S3EncryptionClientException: Re-subscription is not supported! Retry the entire operation. (SDK Attempt Count: 2)
at software.amazon.encryption.s3.S3EncryptionClient.onAbort(S3EncryptionClient.java:359) ~[sd-0.1.jar:?]
at software.amazon.encryption.s3.S3EncryptionClient.multipartPutObject(S3EncryptionClient.java:344) ~[sd-0.1.jar:?]
at software.amazon.encryption.s3.S3EncryptionClient.putObject(S3EncryptionClient.java:196) ~[sd-0.1.jar:?]
... 12 more
Caused by: java.util.concurrent.ExecutionException: software.amazon.encryption.s3.S3EncryptionClientException: Re-subscription is not supported! Retry the entire operation. (SDK Attempt Count: 2)
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at software.amazon.encryption.s3.S3EncryptionClient.multipartPutObject(S3EncryptionClient.java:337) ~[sd-0.1.jar:?]
at software.amazon.encryption.s3.S3EncryptionClient.putObject(S3EncryptionClient.java:196) ~[sd-0.1.jar:?]
... 12 more
Caused by: software.amazon.encryption.s3.S3EncryptionClientException: Re-subscription is not supported! Retry the entire operation. (SDK Attempt Count: 2)
at software.amazon.encryption.s3.internal.UploadObjectObserver$1.call(UploadObjectObserver.java:66) ~[sd-0.1.jar:?]
at software.amazon.encryption.s3.internal.UploadObjectObserver$1.call(UploadObjectObserver.java:56) ~[sd-0.1.jar:?]
at com.alibaba.ttl.TtlCallable.call(TtlCallable.java:65) ~[sd-0.1.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
... 3 more
Caused by: software.amazon.encryption.s3.S3EncryptionClientException: Re-subscription is not supported! Retry the entire operation. (SDK Attempt Count: 2)
at software.amazon.encryption.s3.S3EncryptionClientException$BuilderImpl.build(S3EncryptionClientException.java:67) ~[sd-0.1.jar:?]
at software.amazon.encryption.s3.S3EncryptionClientException$BuilderImpl.build(S3EncryptionClientException.java:44) ~[sd-0.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168) ~[sd-0.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139) ~[sd-0.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157) ~[sd-0.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117) ~[sd-0.1.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78) ~[sd-0.1.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108) ~[sd-0.1.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255) ~[sd-0.1.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167) ~[sd-0.1.jar:?]
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) ~[?:?]
... 3 more
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Write timed out
```
Environment:
Component: AWS S3 Encryption Client (High-level Multipart Upload)
SDK Version: software.amazon.encryption.s3: "3.3.5", software.amazon.awssdk: "2.31.67"
**Network Conditions: Outgoing traffic on port 443 is throttled to 1 Mbps using tc rules.**
Data Source: InputStream (Strict requirement)
Observed behaviour:
1. Due to the 1 Mbps throttle, the upload of a part occasionally times out or disconnects.
2. The AWS SDK correctly identifies the failure and attempts a retry (indicated by SDK Attempt Count: 2 in logs).
3. The retry logic fails immediately because the S3 Encryption Client is unable to reset or re-subscribe to the underlying InputStream.
4. Code snippets:
```
S3EncryotionClient:
S3EncryptionClient
.builder()
.credentialsProvider(...)
.wrappedClient(unencryptedS3Client)
.dualstackEnabled(shouldUseDualStack)
.overrideConfiguration(clientConfigs)
.enableMultipartPutObject(enableHighLevelMultipartUpload)
.enableLegacyUnauthenticatedModes(true)
.enableLegacyWrappingAlgorithms(true)
.enableDelayedAuthenticationMode(true)
.cryptoProvider(cryptoComplyFipsProvider)
unencryptedS3Client:
S3Client
.builder()
.credentialsProvider(
credentials.getAwsV2CredentialsProvider
)
.forcePathStyle(usePathStyleAccess)
.dualstackEnabled(shouldUseDualStack)
.httpClientBuilder(httpClient)
.overrideConfiguration(clientConfigs)
.endpointOverride(s3UriEndpoint)
.region(Region.of(s3Region))
.build()
httpClient:
ApacheHttpClient
.builder()
.useIdleConnectionReaper(useApacheIdleConnectionReaper)
.maxConnections(awsConf.awsMaxConnections.get)
.connectionTimeout(Duration.ofMillis(1000000))
.socketTimeout(Duration.ofMillis(1000000))
.connectionMaxIdleTime(Duration.ofMillis(1000000))
```
### Solution:
It seems sdk is no honouring the timeouts given in httpClient and instead sdk is internally creating a wrapped async client which have its own timeout values.
The above claim is supported by the user agent value found in multipart upload request:
```
POST https://bucket-rsa-akshay-126.s3.us-west-2.amazonaws.com/5gbfile__metadata?uploads
User-Agent=[APN/1.0 Rubrik/1.0 Thor/1.0 aws-sdk-java/2.31.67 md/io#async md/http#NettyNio ...]
while on the other hand head request headers:
HEAD https://bucket-rsa-akshay-126.s3.us-west-2.amazonaws.com/5gbfile
Host=[bucket-rsa-akshay-126.s3.us-west-2.amazonaws.com], User-Agent=[APN/1.0 Rubrik/1.0 Thor/1.0 aws-sdk-java/2.31.67 md/io#sync md/http#Apache...]
```
if async client is mandatory for s3 client it should be restricted if not then encryption client should always use the ApacheHttpClient.
Contributor guide
Research direction
Start with S3EncryptionClient.java around lines 196-198 and 337-359, then inspect UploadObjectObserver.java around lines 56-66 to trace multipart retries for InputStream sources. Compare the sync Apache client configuration with the async client used by multipart upload. Done means a throttled or disconnected multipart upload can retry an InputStream source without the re-subscription failure, with regression coverage for the observed path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100