aws / aws/bedrock-agentcore-starter-toolkit
[BUG] Infinite loop of Cognito M2M and Bedrock requests after enabling Observability
- Dominant language
- Python
- Stars
- 508
- Forks
- 155
- Avg merge
- 8h 50m
- Merged PRs (30d)
- 4
Description
**Describe the bug**
After creating a production-ready agent using the starter toolkit (Terraform + Strands Agents SDK) and configuring observability following the official documentation, I observed a critical issue.
Approximately one day after deployment and invoking the runtime via the Agent Sandbox (invoke ~35 times), CloudTrail logs showed a massive volume of Cognito M2M requests (~2000 Requests) and Bedrock requests. This appears to be an infinite loop, resulting in significant unexpected billing costs.
**To Reproduce**
It might not be able to be reproduced, but this is what I did:
1. Initialize a new project using `agentcore create` with the following selections:
```text
How would you like to start?
> A production-ready agent defined with Terraform or CDK
What agent framework should we use?
> Strands Agents SDK
Which model provider will power your agent?
> Amazon Bedrock
Which IaC provider will define your AgentCore resources?
> Terraform
Initialize a new git repository?
> Yes
```
2. Configure observability by following the guide: [Amazon Bedrock AgentCore Observability Configuration](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html).
3. Deploy the resources using Terraform.
4. Invoke the agent runtime using the Agent Sandbox.
5. Wait for a period of time (approx. 24 hours).
6. Observe CloudTrail for a spike in `Cognito` (M2M token requests) and `Bedrock` API calls.
**Expected behavior**
The agent should process the invocation and remain idle or log telemetry data without triggering further invocations. Enabling observability should not cause a recursive loop of self-invocation or continuous authentication requests.
**Impact**
- **High Severity:** This bug causes an infinite loop of API calls.
- **Financial Impact:** Generated a large unexpected bill due to continuous Cognito and Bedrock usage.
**Error Output**
There is no explicit error output in the console. The issue is observed via CloudTrail logs showing repetitive:
- Cognito M2M token requests (token_POST)
- Cost Explorer :(
**Environment:**
- **Python version:** 3.10.0
- **Package versions:**
- `bedrock-agentcore`: 1.1.2
- `strands-agents`: 1.21.0
Contributor guide
Research direction
Start with the agentcore create production-ready Terraform template, then compare its observability configuration with the Amazon Bedrock AgentCore Observability Configuration guide. Reproduce the deployment and Agent Sandbox invocation if possible, and inspect CloudTrail for repeated Cognito token_POST and Bedrock calls over time. Done means identifying the source of the repeated requests and confirming that an observability-enabled agent remains idle after an invocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, terraform
- Domain
- backend, cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100