aws / aws/agentcore-cli

AgentCore CLI enables OTEL tracing that can create many CloudWatch metrics (increasing cost) without disclosure

Open
#1,472 0 comments 0 reactions 0 assignees View on GitHub
bug documentation
Dominant language
TypeScript
Stars
283
Forks
95
Avg merge
1d 2h
Merged PRs (30d)
183

Description

### Description

When AgentCore Observability enables CloudWatch Transaction Search, it routes X-Ray spans to /aws/application-signals/data. CloudWatch Application Signals then automatically generates metric time series. Because it pre-generates metrics for every combination of API call made x 4 metric names (Latency, Error, Fault, Throttle), this can create many metrics, which are charged per month, and can be costly for agents that use many AWS APIs. There is no documented way to suppress metrics while leaving tracing, although removing the permission from the CloudWatch Logs resource for [xray.amazonaws.com](http://xray.amazonaws.com/) to write to `/aws/application-signals/data` while keeping `aws/spans` for tracing appears to work as an effective workaround.

### Steps to Reproduce

The included zip file contains a reproduction that creates a new agent using the agentcore CLI, replaces its code with calls to list DynamoDB tables and S3 buckets and a second routine that lists only EventBridge subscriptions, runs the first, examines the created metrics (waiting for them to propagate), applies the workaround by removing the CloudWatch Logs resource for [xray.amazonaws.com](http://xray.amazonaws.com/) to write to `/aws/application-signals/data` , then invokes the agent's second routine and shows that the second set of metrics are not logged (but the traces are available).

[metrics-repro.zip](https://github.com/user-attachments/files/28658970/metrics-repro.zip)

### Expected Behavior

Ideally the user does not want to create the expanded metrics (EMF), they only need the traces for AgentCore observability to work. Note this is an issue in X-Ray/CloudWatch Application Signals that requires additional coordination between the AgentCore runtime and X-Ray (X-Ray invokes CloudWatch Application Signals to form enriched metrics itself; AgentCore does not have, or does not invoke, a mechanism to suppress this. Since this can be costly, the AgentCore documentation should warn about Application Signals metric costs when enabling observability and the AgentCore CLI should provide without cost disclosure and require user permission when enabling Application Signals.

### Actual Behavior

The AgentCore CLI automatically configures the AgentCore runtime for observability, which exports OTEL metrics, but also results in those traces being expanded combinatorily to a large set of metrics (currently billed at $.30/month each) and increases the cost to run the agent.

### CLI Version

0.15.0

### Operating System

macOS

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.