awslabs / awslabs/agent-builder-toolkit-aws-transform

AgentRuntimeServer event loop deadlocks on synchronous tool execution, breaking AgentCore /ping health checks

Open
#69 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
4
Avg merge
1d 6h
Merged PRs (30d)
2

Description

In `AgentRuntimeServer` and `BaseServer` regarding how the underlying FastAPI/Uvicorn asynchronous event loop handles long-running synchronous tool processing.

When a custom tool executes synchronously (e.g., waiting for an external data API or polling an asset matrix), it blocks the application’s primary event thread. Because the thread is frozen, the server cannot process the asynchronous `GET /ping` route required by the hosting platform's health check monitoring or the AWS Bedrock AgentCore specification. This triggers aggressive infrastructure timeouts, forcing the container cluster manager to violently issue a `SIGKILL` and spin up new instances mid-execution.

Furthermore, the default implementation of the `/ping` route fails to comply with the official AgentCore specification, resulting in unexpected resource churning and cold starts even when the server is idle.

**1. Synchronous Code Path Execution via the Main Event Loop**

In `agent_runtime_server.py`, the SDK natively executes synchronous agent factories and handles synchronous tool decorators directly inside the main thread loop.
The SDK framework does not offload standard synchronous execution payloads using `asyncio.to_thread` or an execution pool. If a developer uses a traditional `def my_tool()` that performs long-polling or blocks the thread (e.g., using `time.sleep()`), the entire FastAPI event loop freezes.

**2. Event Loop Blockage Halts /ping Heartbeats**

While a blocking tool is processing, the event loop is entirely unavailable. In `base_server.py`, the health probe is registered as an async route.
Because the loop is deadlocked by the running tool, it cannot respond to network probes arriving at /ping. The container fails successive health checks and is repeatedly terminated mid-flight.

**3. Violation of the AWS Bedrock AgentCore Contract**

The `/ping` route implementation is incomplete. According to the internal code documentation and the official AWS Bedrock AgentCore Specification, the response payload requires a tracking timestamp.
The actual return statement completely omits `time_of_last_update`. Additionally, an idle server constantly reports `HealthyBusy` (since the queue worker task is technically always running). Without the timestamp, and with a persistent "Busy" status, the external engine flags the instance as stalled and constantly reaps/restarts idle containers.

Contributor guide

Open the contributing guide

Research direction

Start in agent_runtime_server.py and base_server.py, tracing synchronous agent and tool execution alongside the async GET /ping route. Reproduce a long-running synchronous tool call, then compare health-check behavior and the response payload with the AWS Bedrock AgentCore specification. Done means /ping remains usable during processing and reports the required timestamp and correct idle status.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, fastapi, python
Domain
api, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.