awslabs / awslabs/aws-crt-java
CRT S3 client hangs with concurrent AsyncRequestBody.fromPublisher() uploads in containerized environments
- Dominant language
- Java
- Stars
- 76
- Forks
- 46
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
Reporting two distinct deadlock bugs with the CRT S3 client, originally filed by a customer in [aws-sdk-java-v2#6724](https://github.com/aws/aws-sdk-java-v2/issues/6724). Both have been independently reproduced on our end with Java test cases.
## Issue 1: Event loop thread starvation in low-CPU / container environments
`EventLoopGroup` defaults to `Math.max(1, Runtime.getRuntime().availableProcessors())` threads. In containerized environments with CPU limits (e.g., Kubernetes pods with `cpu: 2000m`), `availableProcessors()` returns 1-2. With a single event loop thread, concurrent uploads via `AsyncRequestBody.fromPublisher()` deadlock — CRT can only process one `sendRequestBody` callback at a time, so if two publishers are interdependent (e.g., `publish().autoConnect(2)`), only one gets subscribed and the other never starts.
**Workaround:** `EventLoopGroup.setStaticDefaultNumThreads(4)` before creating the client.
## Issue 2: Same-client concurrent download+upload hang
Using the same CRT client instance to pipe a download into an upload via `fromPublisher()` hangs silently after N cycles (typically 1-7). This is independent of thread count — reproduces with 4+ event loop threads on a full-CPU machine.
**Trigger:** Same CRT client instance handling a concurrent upload (waiting for body data via `fromPublisher()`) while a download is initiated on that same client.
**Customer's findings** (from their investigation in the issue comments):
| Scenario | Result |
|----------|--------|
| Same CRT client, upload initiated first, download triggered by body publisher | **HANGS** |
| Same CRT client, download completes first, then upload | Passes |
| Two separate CRT client instances (one download, one upload) | Passes |
| CRT download + standard (Netty) upload, same concurrent pattern | Passes |
**Reactive Streams instrumentation** shows where the flow stops on the failing cycle:
```
[upload-body] subscribe() — CRT subscribes to upload body (AwsEventLoop thread)
[upload-body] onSubscribe()
[upload-body] request(1) — CRT requests first chunk...
— nothing else happens. Download body never gets subscribed to.
```
On successful cycles, `request(1)` propagates and triggers the download. On the failing cycle, CRT accepts the download request internally but never delivers the response callback.
**Thread dumps at hang time:** All `AwsEventLoop` threads are `RUNNABLE` (native busy-polling), all `sdk-async-response` threads are `TIMED_WAITING` (idle in `LinkedBlockingQueue.poll`). No Java-level deadlock.
**CRT trace log at hang point:**
```
Requests-in-flight(approx/exact):6/6 Requests-preparing:6 Requests-queued:0
Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0
Requests-streaming-response:0
```
6 requests stuck in "preparing", none on the network, ~3 second gap with zero progress before the log goes quiet.
**Java SDK-side analysis:** The SDK adapter code path (`AsyncRequestBody.fromPublisher()` → `S3CrtRequestBodyStreamAdapter` → `ByteBufferStoringSubscriber`) is non-blocking — no locks or blocking calls.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
Both concurrent S3 uploads should complete successfully regardless of the CPU count or containerization.
### Current Behavior
In environments with low `availableProcessors()` (specifically 1):
- Application hangs indefinitely with no progress
- Only one upload's publisher gets subscribed to by the CRT
- The second upload never begins processing
- No exceptions or timeouts occur — completely silent hang
Works correctly in environments with 3+ processor counts.
### Reproduction Steps
Java reproduction project attached: [6724.zip](https://github.com/user-attachments/files/26101370/6724.zip)
Prerequisites: update BUCKET constant in both test files
Issue 1 — run with -XX:ActiveProcessorCount=1: `./gradlew testBug1`
Issue 2 — run with default processor count: `./gradlew testBug2`
Issue 1 times out after 10s. Issue 2 hangs after 1-7 cycles typically.
Customer's original Kotlin reproduction is also available at [aws-sdk-java-v2#6724 (comment)](https://github.com/aws/aws-sdk-java-v2/issues/6724#issuecomment-4012662409).
### Possible Solution
Calling `EventLoopGroup.setStaticDefaultNumThreads(4)` before creating the S3 client works around the issue.
We're not sure if this is the recommended approach or if there's a better pattern for concurrent reactive uploads with the CRT client in low-CPU environments.
### Additional Information/Context
Related issues we found:
- https://github.com/awslabs/aws-crt-java/issues/656 — Similar symptom but with blocking I/O
- https://github.com/aws/aws-sdk-java-v2/issues/4305 — Buffer blocking (resolved), different from our case
- https://github.com/awslabs/aws-crt-java/issues/836 — Event loop thread discussion
In Kubernetes, CPU limits are commonly used for resource management. A pod with `cpu: 2000m` reports `availableProcessors() = 2`, leading to very few event loop threads.
We'd appreciate any guidance on the recommended pattern for this use case.
### aws-crt-java version used
0.43.4
### Java version used
2.42.15
### Operating System and version
Kubernetes pods with CPU limits (reproduces locally with `-XX:ActiveProcessorCount=1` - macOs Tahoe 26.2)
Contributor guide
Assessment
This issue has not been assessed yet.