awslabs / awslabs/agentcore-samples
Add quality drift detection for the Market Trends Agent's evaluator scores
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
### Summary
The Market Trends Agent use case (`02-use-cases/01-conversational-agents/market-trends-agent/`)
already scores itself with 8 evaluators through an AgentCore online evaluation config, but
nothing watches whether those scores get worse over time. This proposes adding a quality
drift detection feature for that sample.
### What it adds
- A scheduled Lambda that reads evaluator scores from AgentCore Evaluations results,
tracks each evaluator's score stream against its own learned baseline, and raises a
CloudWatch alarm naming the specific evaluator that degraded and stayed degraded.
- Per-evaluator detection method chosen from the measured shape of that evaluator's score
stream (EWMA, z-score, or CUSUM), because a single method applied uniformly gets most of
this agent's evaluators wrong — 5 of 8 produce near-constant scores on healthy traffic.
- Deploy/teardown scripts, a `watch.py` CLI to inspect detector state, a `shape_report.py`
tool to validate the configured method against observed data, and drift-inducing triggers
(`induce_drift.py`) to demo the detector without waiting for real production drift.
- A couple of small fixes surfaced along the way: an evaluator IAM permissions policy that
had unscoped `Resource: "*"` on some log actions, and a trust-policy ARN condition that
didn't cover the runtime's own source ARN.
### Why this, not the existing optimization loop
AgentCore's optimization workflow (`optimization/optimize_agent.py` in this sample) compares
two variants concurrently (A/B) or against a validation batch. That's the wrong tool for
"has this same configuration gotten worse over time," since both arms of an A/B test degrade
together and the comparison reports nothing. This feature tracks one configuration against
its own persisted history instead. Full reasoning is in the feature's own README.
### User experience
**Before:** No signal exists when the agent's live quality degrades (e.g. after a model
version update with no corresponding deploy). Degradation is only found via manual poking
around Evaluations dashboards or when a user complains.
**After:** A CloudWatch alarm fires per evaluator, naming which score stream drifted, with a
dashboard showing pressure toward the alarm threshold, samples seen, and baseline mean per
evaluator.
### Scope
Self-contained under `market-trends-agent/drift_detection/`. Does not modify the agent's
runtime behavior except for two demo-only drift triggers gated behind env vars
(`STALE_PRICES`, `SKIP_PROFILE_STEP`) that are no-ops unless explicitly set.
Contributor guide
Research direction
Start with the Market Trends Agent directory and the feature's README, then read optimization/optimize_agent.py to understand the existing evaluation workflow. Review the proposed drift_detection/ tools, including watch.py, shape_report.py, and induce_drift.py. Done means scheduled detection, per-evaluator CloudWatch alarms and dashboard reporting, deploy/teardown scripts, and the stated IAM and trust-policy fixes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ai, cli, cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100