InvokeModelWithBidirectionalStreamAsync() can block the calling thread forever when the peer never sends any response - no timeout at any layer
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 1.2k
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 12
Description
## Summary
`BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync()` is documented and named as an asynchronous method, but part of the initial event-stream write work runs **synchronously on the calling thread**, inside the call itself, with **no timeout anywhere in the chain** down to raw HTTP/2 flow control. If the server accepts the connection/stream but never sends any response (and therefore never sends a `WINDOW_UPDATE` frame either), the calling thread can hang forever inside the "async" call, not merely leave a callback unfired.
## Environment
- aws-sdk-cpp version: **1.11.890** (built with the CRT HTTP client, `AWS_SDK_USE_CRT_HTTP` / push-based `WriteData` path)
- Service: Bedrock Runtime, `InvokeModelWithBidirectionalStreamAsync` (used for Nova Sonic real-time voice streaming)
- Platform: Windows, MSVC, x86/Win32
- Underlying CRT: aws-crt-cpp (commit pinned by aws-sdk-cpp 1.11.890) / aws-c-http (commit pinned by that aws-crt-cpp)
## Repro
1. Stand up a local HTTP/2 (TLS, self-signed cert is fine) server that:
- Accepts the TCP/TLS/HTTP2 connection and the incoming stream normally.
- Never sends any `WINDOW_UPDATE` frame beyond the connection/stream defaults, and never sends any response (no headers, no data).
2. Call `BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync()` against it (dummy credentials are fine - the server never needs to validate auth), with a `streamReadyHandler` that sends a normal small sequence of bidirectional-streaming events (e.g. a Nova Sonic `sessionStart`/`promptStart`/`contentStart`/`textInput`/`contentEnd`/`promptEnd`/`sessionEnd` sequence) and then calls `Close()` on the input stream.
3. Place a diagnostic statement immediately after the call to `InvokeModelWithBidirectionalStreamAsync()` in your own code.
**Expected**: the call returns promptly (it's documented/named as async), and only the response-received/outcome callback is left waiting - ideally itself bounded by some timeout, or at least clearly the caller's own responsibility to bound since it's callback-based.
**Actual**: the diagnostic statement never executes. The calling thread is parked indefinitely (confirmed via process inspection: near-zero CPU, threads alive, no progress even after 9+ minutes in one run) inside the call itself.
## Root cause (traced through the actual 1.11.890 source)
1. `generated/src/aws-cpp-sdk-bedrock-runtime/source/BedrockRuntimeClient.cpp` (~L401-427), the CRT/push-based branch of `InvokeModelWithBidirectionalStreamAsync()`:
```cpp
auto asyncTask = smithy::client::CreateSmithyBidirectionalWriteDataTask(
this, requestCopy, handler, handlerContext, eventEncoderStream, writeDataStreamBuf, std::move(endpointCallback),
std::move(authCallback));
auto sem = asyncTask.GetSemaphore();
m_clientConfiguration.executor->Submit(std::move(asyncTask));
sem->WaitOne();
streamReadyHandler(*eventEncoderStream);
```
`sem->WaitOne()` is unconditional (no timeout). Once it returns (i.e. once the background task's `HttpWriteDataStreamBuf::Initialize()` succeeds), `streamReadyHandler` - the caller-supplied write callback - is invoked **directly on the calling thread**, as the last statement of this method. There is nothing else in the function body after this call; the method returns only once `streamReadyHandler` returns.
2. Inside `streamReadyHandler`, calling `Close()` on the event-stream input (`Model::InvokeModelWithBidirectionalStreamInput` → inherited `Aws::Utils::Event::EventEncoderStream::Close()`) is a pure passthrough down to `HttpWriteDataStreamBuf::Close()` → `SendBuffer(endStream=true)`. The same `SendBuffer()` is used for every write (`overflow()`, `xsputn()`, `sync()`, and `Close()`).
3. `src/aws-cpp-sdk-core/source/utils/stream/HttpWriteDataStreamBuf.cpp` (~L148-181), `SendBuffer()`:
```cpp
m_stream->WriteData(data, [this](int errorCode) -> void {
std::unique_lock const lock{m_writeMutex};
m_writeInProgress = false;
m_writeError = (errorCode != AWS_ERROR_SUCCESS);
m_writeComplete.notify_one();
}, endStream);
std::unique_lock lock{m_writeMutex};
m_writeComplete.wait(lock, [this]() -> bool { return !m_writeInProgress; });
```
This wait has **no timeout parameter and no timed overload anywhere in the class**.
4. The completion callback only fires once aws-c-http's `aws_h2_stream_encode_data_frame()` (`source/h2_stream.c`, ~L829-846) actually encodes the write into an outgoing DATA frame:
```c
if (stream->thread_data.window_size_peer <= AWS_H2_MIN_WINDOW_SIZE) {
/* The stream is stalled now */
*data_encode_status = AWS_H2_DATA_ENCODE_ONGOING_WINDOW_STALLED;
return AWS_OP_SUCCESS;
}
```
If the stream is window-stalled, this returns having done **nothing** - no error, no retry scheduling, no deadline. The queued write just sits there.
5. `window_size_peer` is this side's outbound HTTP/2 flow-control credit as granted by the peer - seeded from the RFC 7540 §6.5.2 default (65535 bytes) at connection/stream setup (`source/h2_connection.c` ~L387-388, `source/h2_stream.c` ~L741-745), and only ever replenished by a `WINDOW_UPDATE` frame **from the peer**. A peer that never sends anything never sends one either. Once cumulative outstanding bytes across the whole stream exceed that window, every subsequent write - including, in this repro, the one inside `Close()` - stalls forever.
6. The one native mechanism that exists specifically for "peer never responds" is `response_first_byte_timeout_ms` (`aws-c-http`'s `aws_http_connection_manager_options` / `aws_http_make_request_options`), but per its own doc comment it is explicitly HTTP/1.1-only:
```c
/**
* ...
* TODO: Only supported in HTTP/1.1 now, support it in HTTP/2
*/
uint64_t response_first_byte_timeout_ms;
```
confirmed by grepping the whole `aws-c-http` tree: it's referenced only in `h1_connection.c`/`h1_stream.c`, never in `h2_connection.c`/`h2_stream.c`. It is also not exposed anywhere through `aws-crt-cpp`'s or `aws-sdk-cpp`'s C++ wrapper types, even for HTTP/1.1.
7. Separately, `ClientConfiguration::requestTimeoutMs` exists and is checked in the SDK's synchronous `MakeRequest()` response wait (`src/aws-cpp-sdk-core/source/http/crt/CRTHttpClient.cpp` ~L537-559), but is never referenced anywhere in the connection-acquisition (`AcquireConnection`, ~L721-745) or bidirectional-streaming write path - so it provides no protection here even when configured.
## Impact
For a caller of `InvokeModelWithBidirectionalStreamAsync()`, there is currently no supported way, at any layer, to bound how long the call itself (not just the eventual response callback) can block against a peer that accepts the stream but never communicates further. This is worse than "the outcome callback never fires" - the calling application's own thread is the one that hangs, which for an application built expecting an async, callback-driven API (e.g. a call-handling engine dispatching this from a request-processing thread) can freeze application logic entirely, not just leave a background task dangling.
## Suggested areas to address (not prescribing the fix, just where the gaps are)
- Expose `response_first_byte_timeout_ms` (or an HTTP/2-appropriate equivalent) through `aws-crt-cpp` and `aws-sdk-cpp`'s `ClientConfiguration`, and implement it for HTTP/2 in `aws-c-http` (currently HTTP/1.1-only by an existing TODO).
- At minimum, give `HttpWriteDataStreamBuf::SendBuffer()`'s write-completion wait a timeout path, so a stalled write surfaces as an error to the caller instead of blocking forever.
- Consider whether `streamReadyHandler` needs to run synchronously on the calling thread inside `InvokeModelWithBidirectionalStreamAsync()` at all - if the intent is a genuinely async method, dispatching this initial write burst through the executor (like the rest of the task) rather than back on the caller's thread would preserve the documented async contract even without a new timeout feature.
## Related, but distinct issues (searched, not duplicates)
- #3917 / #3911 - `Aws::ShutdownAPI()` hang after a stream has already *completed* (self-reference cycle pinning a connection) - already fixed in PR #3919 (1.11.890). This report is a different bug: the hang happens *during* the initial call, before any response, triggered by HTTP/2 flow control against a non-responsive peer, not by a resource cycle at shutdown.
- #3649, #3650 - connectivity/error-visibility issues, not hangs of this kind.
Contributor guide
Research direction
Start with generated/src/aws-cpp-sdk-bedrock-runtime/source/BedrockRuntimeClient.cpp and src/aws-cpp-sdk-core/source/utils/stream/HttpWriteDataStreamBuf.cpp, then trace the CRT HTTP/2 write path through aws-c-http's source/h2_stream.c. Reproduce the stalled peer scenario and inspect the existing timeout handling. Done means the asynchronous call and its initial writes are bounded and the failure is observable rather than hanging indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- api, backend, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100