Seeing Upload failure while uploading object to S3 for OpenSearch service.

### Describe the bug

We are seeing Issue while uploading translog data to S3 while using aws-crt client. The issue seem to be happening when data is being ingested to opensearch service under heavy load. Not seen in normal loads.

We tried the same ingestion load using Netty client and we don't see any failure.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

We should not be seeing any upload failure.

### Current Behavior

We get the below exception.

`025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] java.lang.IllegalStateException: The request content has fewer bytes than the specified content-length: 16912416 bytes.
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.io.SdkLengthAwareInputStream.read(SdkLengthAwareInputStream.java:82)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.FilterInputStream.read(FilterInputStream.java:95)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:55)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:92)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:112)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.lang.Thread.run(Thread.java:1583)`

`[2025-11-07T05:40:34,742][ERROR][o.o.i.s.RemoteDirectory ][55a662f74df1eebcd05dc39abeb4cb07] Failed to upload blob _7s9.cfs
software.amazon.awssdk.core.exception.SdkClientException: Failed to send multipart upload requests.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at org.opensearch.repositories.s3.async.AsyncTransferManager.handleException(AsyncTransferManager.java:333)
at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$handleExceptionOrResponse$13(AsyncTransferManager.java:279)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$allOfExceptionForwarded$5(CompletableFutureUtils.java:195)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$anyFail$6(CompletableFutureUtils.java:217)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:56)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:67)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: A callback has reported failure. (SDK Attempt Count: 2)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
... 26 more
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: software.amazon.awssdk.crt.http.HttpException: The connection has closed or is closing.
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: software.amazon.awssdk.crt.http.HttpException: The connection has closed or is closing.
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when acquiring a connection
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request
Caused by: software.amazon.awssdk.crt.http.HttpException: A callback has reported failure.
at software.amazon.awssdk.http.crt.internal.response.CrtResponseAdapter.onResponseComplete(CrtResponseAdapter.java:119)`

`[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.StoringSubscriber$Event.runtimeError(StoringSubscriber.java:181)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.ByteBufferStoringSubscriber.transferTo(ByteBufferStoringSubscriber.java:114)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.http.crt.internal.request.CrtRequestBodyAdapter.sendRequestBody(CrtRequestBodyAdapter.java:43)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] Caused by: java.io.IOException: Stream closed
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:188)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:198)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:352)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.io.SdkLengthAwareInputStream.read(SdkLengthAwareInputStream.java:75)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.FilterInputStream.read(FilterInputStream.java:95)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:55)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:92)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:112)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.lang.Thread.run(Thread.java:1583)`

### Reproduction Steps

Load the data in Opensearch service using multiple threads.

### Possible Solution

NA

### Additional Information/Context

Related OpenSearch PR: https://github.com/opensearch-project/OpenSearch/pull/18800

### aws-crt-java version used

0.35.0

### Java version used

JAVA21

### Operating System and version

Linux