aws / aws/aws-sdk-java-v2

ParallelMultipartDownloaderSubscriber.onError cancels part futures before completing resultFuture, swallowing the original error

Open Beginner friendly
#7,174 0 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the bug

When a multipart download (Netty-based `S3AsyncClient` with `multipartEnabled(true)`, via `S3TransferManager.downloadFile`) fails, the caller's `completionFuture()` completes with a bare `java.util.concurrent.CancellationException` that has no cause attached. The actual `Throwable` that triggered the
failure is unrecoverable by the application.

Root cause is the ordering in `ParallelMultipartDownloaderSubscriber.onError` (`services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java`):

@Override
public void onError(Throwable t) {
inFlightRequests.values().forEach(future -> future.cancel(true)); // 1. cancel first
inFlightRequests.clear();
resultFuture.completeExceptionally(t); // 2. real cause last
}

The `future.cancel(true)` calls propagate a `CancellationException` through the transformer chain (`FileAsyncResponseTransformerPublisher`) to the future the caller observes, before `resultFuture.completeExceptionally(t)` runs. The method also does not log `t`, so the trigger is invisible even at
DEBUG level.

Related earlier report: #6612 (closed for staleness without a fix).

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

The caller-visible future should fail with the original `Throwable` that triggered `onError` (or at minimum a `CancellationException` whose cause is `t`), and `t` should be logged, so applications can classify and handle the real failure.

### Current Behavior

The caller's `completionFuture()` fails with a bare `CancellationException` with no cause and no suppressed exceptions. Stack trace observed in production:

java.util.concurrent.CancellationException
at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2478)
at software.amazon.awssdk.services.s3.internal.multipart.ParallelMultipartDownloaderSubscriber.lambda$onError$18(ParallelMultipartDownloaderSubscriber.java:418)
at java.base/java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4780)
at software.amazon.awssdk.services.s3.internal.multipart.ParallelMultipartDownloaderSubscriber.onError(ParallelMultipartDownloaderSubscriber.java:418)
at software.amazon.awssdk.utils.internal.MappingSubscriber.onError(MappingSubscriber.java:60)
at software.amazon.awssdk.core.internal.async.FileAsyncResponseTransformerPublisher$IndividualFileTransformer.onResponse(FileAsyncResponseTransformerPublisher.java:111)
at software.amazon.awssdk.core.async.listener.AsyncResponseTransformerListener$NotifyingAsyncResponseTransformer.onResponse(AsyncResponseTransformerListener.java:92)
at software.amazon.awssdk.core.internal.http.async.AsyncStreamingResponseHandler.onHeaders(AsyncStreamingResponseHandler.java:55)
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelRead0(ResponseHandler.java:101)
...

Note there is no `Caused by:` — the original error that triggered `onError` is lost. No log line is emitted by the subscriber either, so the trigger cannot be recovered even with SDK DEBUG logging enabled.

### Reproduction Steps

Any failure injected into an in-flight multipart download reproduces the cause-swallowing. The simplest deterministic repro is to close the client while a large download is in flight (the terminated scheduled executor rejects a part retry, which triggers `onError`, but any part-level failure takes
the same path):

S3AsyncClient s3 = S3AsyncClient.builder()
.region(Region.US_EAST_1)
.multipartEnabled(true)
.build();
S3TransferManager tm = S3TransferManager.builder().s3Client(s3).build();

FileDownload download = tm.downloadFile(DownloadFileRequest.builder()
.getObjectRequest(b -> b.bucket("").key(""))
.destination(Paths.get("/tmp/out.bin"))
.build());

// Induce a failure mid-download, e.g. close the client while parts are in flight
Thread.sleep(500);
tm.close();
s3.close();

try {
download.completionFuture().join();
} catch (CompletionException e) {
Throwable cause = e.getCause();
System.out.println(cause); // java.util.concurrent.CancellationException
System.out.println(cause.getCause()); // null <-- original trigger lost
}

Expected: the future fails with the underlying error (here a `RejectedExecutionException` from the terminated executor), or a `CancellationException` carrying it as cause. Actual: a bare `CancellationException`, cause `null`.

### Possible Solution

This is already fixed in the sibling class in the same package. `ParallelPresignedUrlMultipartDownloaderSubscriber.onError` completes `resultFuture` before cancelling, and logs the error:

@Override
public void onError(Throwable t) {
log.debug(() -> "Error in parallel multipart download", t);
resultFuture.completeExceptionally(t);
inFlightRequests.values().forEach(future -> future.cancel(true));
}

Its `resultFuture` field Javadoc documents the reasoning: "Completed exceptionally on error (before cancel)...". Applying the same ordering and logging to `ParallelMultipartDownloaderSubscriber` resolves this:

@Override
public void onError(Throwable t) {
log.debug(() -> "Error in parallel multipart download", t);
resultFuture.completeExceptionally(t);
inFlightRequests.values().forEach(future -> future.cancel(true));
inFlightRequests.clear();
}

### Additional Information/Context

Observed in a production service performing concurrent large-file multipart downloads. Failures arrive in clusters; because no cause survives, the application cannot distinguish transient client-side conditions from real S3 errors, forcing misclassification. We separately confirmed via a
request-level interceptor that at least one trigger cohort is a `RejectedExecutionException` from a terminated scheduled executor, but that evidence is only available at the HTTP layer — the transfer-level future discards it.

### AWS Java SDK version used

AWS Java SDK version: 2.x (Netty async client; bug present in current master per cited source)

### JDK version used

JDK: 17

### Operating System and version

Amazon Linux 2 (x86_64)

Contributor guide

Open the contributing guide

Research direction

Start in services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java, focusing on onError and its resultFuture ordering. Compare it with ParallelPresignedUrlMultipartDownloaderSubscriber.onError, then verify that an injected multipart-download failure preserves the original Throwable and emits the described debug log.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.