[BUG] snapshot restore on remote-backed cluster never finishes

Describe the bug

I'm trying to restore snapshot from s3 repository with remote-backed storage configured, but it doesn't work.

I have a lot of java.lang.WrongThreadException: Attempted access outside owning thread errors in the logs:

2025-03-07T15:29:13.459Z | [2025-03-07T15:29:13,456][WARN ][o.o.i.s.RemoteStoreRefreshListener] [i-0ed41ec745a2bc21b] [index_1][0] Exception while uploading new segments to the remote segment store
-- | --
  | 2025-03-07T15:29:13.459Z | java.util.concurrent.CompletionException: java.lang.WrongThreadException: Attempted access outside owning thread
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:936) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) [utils-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:54) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:67) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) [utils-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) [utils-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) [?:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:240) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163) [sdk-core-2.20.86.jar:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) [?:?]
  | 2025-03-07T15:29:13.459Z | at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:946) [opensearch-2.18.0.jar:2.18.0]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
  | 2025-03-07T15:29:13.459Z | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Related component

Storage:Snapshots

To Reproduce

1. Deploy Opensearch cluster with remote-backed storage configured
2. Try to restore any snapshot into it
3. See how it stuck in FINALIZED phase and never finish
4. See java.lang.WrongThreadException: Attempted access outside owning thread in Opensearch logs

Expected behavior

Snapshot being successfully restored to the cluster.

Additional Details

Plugins
ec2-discovery
s3-repository

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: public.ecr.aws/opensearchproject/opensearch
• Version 2.18.0, 2.19.0, 2.19.1

Additional context
I also see IncompleteBody: The request body terminated unexpectedly in cloudtrail for s3:PutObject operations.

Review the snapshot restore flow and the RemoteStoreRefreshListener named in the stack trace; inspect how asynchronous S3 PutObject failures are handled. Reproduce the issue with remote-backed storage and a snapshot restore, then verify restoration no longer remains in FINALIZED and the reported WrongThreadException and IncompleteBody errors are absent.