ParallelMultipartDownloaderSubscriber.onError cancels part futures before completing resultFuture, swallowing the original error
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 51
Description
### Describe the bug
When a multipart download (Netty-based `S3AsyncClient` with `multipartEnabled(true)`, via `S3TransferManager.downloadFile`) fails, the caller's `completionFuture()` completes with a bare `java.util.concurrent.CancellationException` that has no cause attached. The actual `Throwable` that triggered the
failure is unrecoverable by the application.
Root cause is the ordering in `ParallelMultipartDownloaderSubscriber.onError` (`services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java`):
@Override
public void onError(Throwable t) {
inFlightRequests.values().forEach(future -> future.cancel(true)); // 1. cancel first
inFlightRequests.clear();
resultFuture.completeExceptionally(t); // 2. real cause last
}
The `future.cancel(true)` calls propagate a `CancellationException` through the transformer chain (`FileAsyncResponseTransformerPublisher`) to the future the caller observes, before `resultFuture.completeExceptionally(t)` runs. The method also does not log `t`, so the trigger is invisible even at
DEBUG level.
Related earlier report: #6612 (closed for staleness without a fix).
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
The caller-visible future should fail with the original `Throwable` that triggered `onError` (or at minimum a `CancellationException` whose cause is `t`), and `t` should be logged, so applications can classify and handle the real failure.
### Current Behavior
The caller's `completionFuture()` fails with a bare `CancellationException` with no cause and no suppressed exceptions. Stack trace observed in production:
java.util.concurrent.CancellationException
at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2478)
at software.amazon.awssdk.services.s3.internal.multipart.ParallelMultipartDownloaderSubscriber.lambda$onError$18(ParallelMultipartDownloaderSubscriber.java:418)
at java.base/java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4780)
at software.amazon.awssdk.services.s3.internal.multipart.ParallelMultipartDownloaderSubscriber.onError(ParallelMultipartDownloaderSubscriber.java:418)
at software.amazon.awssdk.utils.internal.MappingSubscriber.onError(MappingSubscriber.java:60)
at software.amazon.awssdk.core.internal.async.FileAsyncResponseTransformerPublisher$IndividualFileTransformer.onResponse(FileAsyncResponseTransformerPublisher.java:111)
at software.amazon.awssdk.core.async.listener.AsyncResponseTransformerListener$NotifyingAsyncResponseTransformer.onResponse(AsyncResponseTransformerListener.java:92)
at software.amazon.awssdk.core.internal.http.async.AsyncStreamingResponseHandler.onHeaders(AsyncStreamingResponseHandler.java:55)
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelRead0(ResponseHandler.java:101)
...
Note there is no `Caused by:` — the original error that triggered `onError` is lost. No log line is emitted by the subscriber either, so the trigger cannot be recovered even with SDK DEBUG logging enabled.
### Reproduction Steps
Any failure injected into an in-flight multipart download reproduces the cause-swallowing. The simplest deterministic repro is to close the client while a large download is in flight (the terminated scheduled executor rejects a part retry, which triggers `onError`, but any part-level failure takes
the same path):
S3AsyncClient s3 = S3AsyncClient.builder()
.region(Region.US_EAST_1)
.multipartEnabled(true)
.build();
S3TransferManager tm = S3TransferManager.builder().s3Client(s3).build();
FileDownload download = tm.downloadFile(DownloadFileRequest.builder()
.getObjectRequest(b -> b.bucket("").key(""))
.destination(Paths.get("/tmp/out.bin"))
.build());
// Induce a failure mid-download, e.g. close the client while parts are in flight
Thread.sleep(500);
tm.close();
s3.close();
try {
download.completionFuture().join();
} catch (CompletionException e) {
Throwable cause = e.getCause();
System.out.println(cause); // java.util.concurrent.CancellationException
System.out.println(cause.getCause()); // null <-- original trigger lost
}
Expected: the future fails with the underlying error (here a `RejectedExecutionException` from the terminated executor), or a `CancellationException` carrying it as cause. Actual: a bare `CancellationException`, cause `null`.
### Possible Solution
This is already fixed in the sibling class in the same package. `ParallelPresignedUrlMultipartDownloaderSubscriber.onError` completes `resultFuture` before cancelling, and logs the error:
@Override
public void onError(Throwable t) {
log.debug(() -> "Error in parallel multipart download", t);
resultFuture.completeExceptionally(t);
inFlightRequests.values().forEach(future -> future.cancel(true));
}
Its `resultFuture` field Javadoc documents the reasoning: "Completed exceptionally on error (before cancel)...". Applying the same ordering and logging to `ParallelMultipartDownloaderSubscriber` resolves this:
@Override
public void onError(Throwable t) {
log.debug(() -> "Error in parallel multipart download", t);
resultFuture.completeExceptionally(t);
inFlightRequests.values().forEach(future -> future.cancel(true));
inFlightRequests.clear();
}
### Additional Information/Context
Observed in a production service performing concurrent large-file multipart downloads. Failures arrive in clusters; because no cause survives, the application cannot distinguish transient client-side conditions from real S3 errors, forcing misclassification. We separately confirmed via a
request-level interceptor that at least one trigger cohort is a `RejectedExecutionException` from a terminated scheduled executor, but that evidence is only available at the HTTP layer — the transfer-level future discards it.
### AWS Java SDK version used
AWS Java SDK version: 2.x (Netty async client; bug present in current master per cited source)
### JDK version used
JDK: 17
### Operating System and version
Amazon Linux 2 (x86_64)
Contributor guide
Research direction
Start in services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java, focusing on onError and its resultFuture ordering. Compare it with ParallelPresignedUrlMultipartDownloaderSubscriber.onError, then verify that an injected multipart-download failure preserves the original Throwable and emits the described debug log.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100