awslabs / awslabs/aws-crt-java

CRT S3 client hangs with concurrent AsyncRequestBody.fromPublisher() uploads in containerized environments

Open
#976 2 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
Java
Stars
76
Forks
46
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

Reporting two distinct deadlock bugs with the CRT S3 client, originally filed by a customer in [aws-sdk-java-v2#6724](https://github.com/aws/aws-sdk-java-v2/issues/6724). Both have been independently reproduced on our end with Java test cases.

## Issue 1: Event loop thread starvation in low-CPU / container environments

`EventLoopGroup` defaults to `Math.max(1, Runtime.getRuntime().availableProcessors())` threads. In containerized environments with CPU limits (e.g., Kubernetes pods with `cpu: 2000m`), `availableProcessors()` returns 1-2. With a single event loop thread, concurrent uploads via `AsyncRequestBody.fromPublisher()` deadlock — CRT can only process one `sendRequestBody` callback at a time, so if two publishers are interdependent (e.g., `publish().autoConnect(2)`), only one gets subscribed and the other never starts.

**Workaround:** `EventLoopGroup.setStaticDefaultNumThreads(4)` before creating the client.

## Issue 2: Same-client concurrent download+upload hang

Using the same CRT client instance to pipe a download into an upload via `fromPublisher()` hangs silently after N cycles (typically 1-7). This is independent of thread count — reproduces with 4+ event loop threads on a full-CPU machine.

**Trigger:** Same CRT client instance handling a concurrent upload (waiting for body data via `fromPublisher()`) while a download is initiated on that same client.

**Customer's findings** (from their investigation in the issue comments):

| Scenario | Result |
|----------|--------|
| Same CRT client, upload initiated first, download triggered by body publisher | **HANGS** |
| Same CRT client, download completes first, then upload | Passes |
| Two separate CRT client instances (one download, one upload) | Passes |
| CRT download + standard (Netty) upload, same concurrent pattern | Passes |

**Reactive Streams instrumentation** shows where the flow stops on the failing cycle:

```
[upload-body] subscribe() — CRT subscribes to upload body (AwsEventLoop thread)
[upload-body] onSubscribe()
[upload-body] request(1) — CRT requests first chunk...
— nothing else happens. Download body never gets subscribed to.
```

On successful cycles, `request(1)` propagates and triggers the download. On the failing cycle, CRT accepts the download request internally but never delivers the response callback.

**Thread dumps at hang time:** All `AwsEventLoop` threads are `RUNNABLE` (native busy-polling), all `sdk-async-response` threads are `TIMED_WAITING` (idle in `LinkedBlockingQueue.poll`). No Java-level deadlock.

**CRT trace log at hang point:**

```
Requests-in-flight(approx/exact):6/6 Requests-preparing:6 Requests-queued:0
Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0
Requests-streaming-response:0
```

6 requests stuck in "preparing", none on the network, ~3 second gap with zero progress before the log goes quiet.

**Java SDK-side analysis:** The SDK adapter code path (`AsyncRequestBody.fromPublisher()` → `S3CrtRequestBodyStreamAdapter` → `ByteBufferStoringSubscriber`) is non-blocking — no locks or blocking calls.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

Both concurrent S3 uploads should complete successfully regardless of the CPU count or containerization.

### Current Behavior

In environments with low `availableProcessors()` (specifically 1):

- Application hangs indefinitely with no progress
- Only one upload's publisher gets subscribed to by the CRT
- The second upload never begins processing
- No exceptions or timeouts occur — completely silent hang

Works correctly in environments with 3+ processor counts.

### Reproduction Steps

Java reproduction project attached: [6724.zip](https://github.com/user-attachments/files/26101370/6724.zip)

Prerequisites: update BUCKET constant in both test files

Issue 1 — run with -XX:ActiveProcessorCount=1: `./gradlew testBug1`

Issue 2 — run with default processor count: `./gradlew testBug2`

Issue 1 times out after 10s. Issue 2 hangs after 1-7 cycles typically.

Customer's original Kotlin reproduction is also available at [aws-sdk-java-v2#6724 (comment)](https://github.com/aws/aws-sdk-java-v2/issues/6724#issuecomment-4012662409).

### Possible Solution

Calling `EventLoopGroup.setStaticDefaultNumThreads(4)` before creating the S3 client works around the issue.

We're not sure if this is the recommended approach or if there's a better pattern for concurrent reactive uploads with the CRT client in low-CPU environments.

### Additional Information/Context

Related issues we found:

- https://github.com/awslabs/aws-crt-java/issues/656 — Similar symptom but with blocking I/O
- https://github.com/aws/aws-sdk-java-v2/issues/4305 — Buffer blocking (resolved), different from our case
- https://github.com/awslabs/aws-crt-java/issues/836 — Event loop thread discussion

In Kubernetes, CPU limits are commonly used for resource management. A pod with `cpu: 2000m` reports `availableProcessors() = 2`, leading to very few event loop threads.

We'd appreciate any guidance on the recommended pattern for this use case.

### aws-crt-java version used

0.43.4

### Java version used

2.42.15

### Operating System and version

Kubernetes pods with CPU limits (reproduces locally with `-XX:ActiveProcessorCount=1` - macOs Tahoe 26.2)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.