aws / aws/agentcore-cli

feat(traces): compare two AgentCore traces from the CLI

Open
#1,830 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
283
Forks
95
Avg merge
1d 2h
Merged PRs (30d)
183

Description

## Problem

`agentcore traces list` finds traces and `agentcore traces get` exports raw
CloudWatch records, but there is no human-friendly way to compare two agent
invocations.

Users testing runtime versions or named endpoints must manually download JSON,
inspect CloudWatch’s incomplete single-span view, and calculate latency/token
differences themselves. This makes performance regression testing impractical.

## Proposed command

```bash
agentcore traces compare \
--runtime \
[--since

Example output:

```
Trace comparison: baseline abc123 → candidate def456

Metric Baseline Candidate Delta
End-to-end latency 10.00s 6.33s -3.66s (-36.6%)
LLM latency 7.36s 3.67s -3.69s (-50.1%)
Tool latency 2.26s 2.58s +0.32s (+14.3%)
LLM calls 2 2 0 (0.0%)
Tool calls 1 1 0 (0.0%)
Input tokens 151,266 2,848 -148,418 (-98.1%)
Output tokens 595 300 -295 (-49.6%)
Total tokens 151,861 3,148 -148,713 (-97.9%)

Baseline model(s): us.anthropic.claude-haiku-4-5-20251001-v1:0
Candidate model(s): us.anthropic.claude-haiku-4-5-20251001-v1:0
```

## Required behavior

- Resolve the runtime/project target using the existing traces command
conventions.

- Fetch structured span records from CloudWatch directly.
- Use the POST /invocations server span for end-to-end latency when present.
If absent, fall back to earliest span start through latest span end and label
that fallback clearly.

- Report LLM and tool time separately using existing GenAI span attributes.
- Avoid double-counting nested provider spans. For example, a Strands internal
LLM span and its nested Bedrock client span represent the same model call.

- Include LLM/tool call counts and input/output/total token counts when
available.

- Show absolute and percentage deltas; handle a zero baseline safely.
- Provide stable machine-readable output with --json.
- Fail clearly if either trace cannot be found or has no usable timed spans.
- Do not claim a “critical path” calculation: CloudWatch trace parent/child
relationships may be incomplete.

- Show comparability warnings when observable characteristics differ, such as
LLM-call count, tool-call count, or token usage. The CLI cannot prove that
## Scope

This is a trace-analysis and latency-comparison feature. It is independent of:

- runtime endpoint selection for agentcore invoke
- AgentCore Gateway A/B-test infrastructure
- SigNoz or other third-party OTEL observability backends

### Acceptance Criteria

- agentcore traces compare works for two trace IDs from the same runtime.
- It produces the table above or equivalent concise terminal output.
- --json returns documented structured data suitable for CI benchmarking.
- Unit tests cover root-span selection, fallback timing, nested-span
de-duplication, token aggregation, deltas, warnings, and missing-trace
errors.

- Existing traces list and traces get behavior remains unchanged.

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the existing `agentcore traces list` and `agentcore traces get` commands, their CloudWatch trace/span sources, target-resolution conventions, and GenAI span attributes. Then use the requested unit-test areas—root-span selection, fallback timing, nested-span de-duplication, token aggregation, deltas, warnings, and missing-trace errors—as the definition of done, while preserving existing list/get behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cli, cloud, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.