aws / aws/bedrock-agentcore-starter-toolkit

No API to list or force-terminate active runtime sessions — runaway agents burn uncapped costs

Open
#498 5 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
508
Forks
155
Avg merge
8h 50m
Merged PRs (30d)
4

Description

# Runaway Agent Issue: AWS Bedrock AgentCore Has No Runtime Session Kill Switch

## Summary

AWS Bedrock AgentCore provides no API to list or force-terminate active runtime sessions (microVM instances). When an agent enters a runaway loop, operators have no direct way to stop it — the agent continues consuming API tokens and incurring costs until it dies naturally or is killed via indirect workarounds.

## Observed Incident

- **Runtime:** `` (dev)
- **Duration:** ~58 minutes
- **Cost incurred:** **$72.67** from a single runaway session
- **Root cause:** A Claude Code `Stop` hook failure produced error output that was fed back as a new prompt on every turn, creating a self-sustaining loop with no exit condition.

## Why It's Hard to Stop

### 1. No API to list active runtime sessions

```bash
$ aws bedrock-agentcore list-sessions
# ERROR: requires --memory-id and --actor-id
# This lists Memory sessions, NOT runtime/microVM sessions
```

There is no equivalent of `kubectl get pods` or `ecs list-tasks` for AgentCore microVMs. The only visibility into running sessions is CloudWatch logs.

### 2. `stop_runtime_session` does not kill the container

```python
client.stop_runtime_session(
agentRuntimeArn='',
runtimeSessionId='',
)
# ResourceNotFoundException: Session not found or has been terminated
```

AgentCore had already marked the session terminated at the control plane, but the underlying microVM container kept running independently and continued making Bedrock API calls.

### 3. `update_agent_runtime` does not kill active containers mid-loop

Calling `update_agent_runtime` (even with a new image URI, as a CI/CD deploy does) does not forcefully terminate containers that are actively processing requests. AgentCore waits for the container to go idle before applying the update — but a looping agent never goes idle.

### 4. Persistent volume backup creates a deadlock

Once a shutdown is initiated, AgentCore tries to back up the persistent volume (`/mnt/workspace`) before killing the container. A looping agent continuously writes to the volume, so the backup never completes and the container never dies:

```
Write failed: waiting to be backed up. ← repeated indefinitely
Claude: OK.
Write failed: waiting to be backed up.
Claude: Noted.
```

## How We Killed It

After exhausting the above options, the only effective solution was to **deny Bedrock API permissions on the runtime's IAM role**:

```python
iam.put_role_policy(
RoleName='',
PolicyName='emergency-bedrock-deny',
PolicyDocument={
"Version": "2012-10-17",
"Statement": [{
"Effect": "Deny",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream",
"bedrock:CallWithBearerToken"
],
"Resource": "*"
}]
}
)
```

This caused the agent's next Bedrock call to fail with a 403, breaking the loop. The IAM change propagated in ~30–60 seconds.

**Side effects:** This also breaks all other active sessions using the same runtime role. Not acceptable for production.

## What AWS Should Provide

1. **`list_runtime_sessions`** — list all active microVM sessions per runtime, with session ID, start time, and invocation count.
2. **`force_stop_runtime_session`** — immediately SIGKILL the container regardless of in-flight requests or pending volume backups.
3. **Per-session spend alerts** — CloudWatch alarm on Bedrock token spend scoped to a single `runtimeSessionId`.
4. **Configurable max-cost / max-turn guardrails** at the AgentCore runtime level, not just in agent code.

## Recommendations for AgentCore Users

Until AWS addresses this, self-protect with:

- Always set `max_turns` in your agent SDK options.
- Never configure a `Stop` hook that can fail and produce output that gets re-ingested as a prompt.
- Set up a CloudWatch metric alarm on your Bedrock invocation costs with a short evaluation period (e.g., >$1 in 5 minutes).
- Keep your runtime IAM role isolated (one role per runtime) so an emergency deny policy doesn't blast other workloads.
- Know your runtime role name ahead of time — you'll need it fast if a runaway starts.

Contributor guide

Open the contributing guide

Research direction

No repository file, test, or entry point is named. Start by checking whether this toolkit exposes AgentCore runtime-session operations, then compare the requested listing, termination, spend-alert, and guardrail capabilities with the AWS API documentation; done means identifying an implementable toolkit change or confirming that the request requires AWS service support.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend-api-design, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.