awslabs / awslabs/agentcore-samples

Add quality drift detection for the Market Trends Agent's evaluator scores

Open
#2,055 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
30

Description

### Summary

The Market Trends Agent use case (`02-use-cases/01-conversational-agents/market-trends-agent/`)
already scores itself with 8 evaluators through an AgentCore online evaluation config, but
nothing watches whether those scores get worse over time. This proposes adding a quality
drift detection feature for that sample.

### What it adds

- A scheduled Lambda that reads evaluator scores from AgentCore Evaluations results,
tracks each evaluator's score stream against its own learned baseline, and raises a
CloudWatch alarm naming the specific evaluator that degraded and stayed degraded.
- Per-evaluator detection method chosen from the measured shape of that evaluator's score
stream (EWMA, z-score, or CUSUM), because a single method applied uniformly gets most of
this agent's evaluators wrong — 5 of 8 produce near-constant scores on healthy traffic.
- Deploy/teardown scripts, a `watch.py` CLI to inspect detector state, a `shape_report.py`
tool to validate the configured method against observed data, and drift-inducing triggers
(`induce_drift.py`) to demo the detector without waiting for real production drift.
- A couple of small fixes surfaced along the way: an evaluator IAM permissions policy that
had unscoped `Resource: "*"` on some log actions, and a trust-policy ARN condition that
didn't cover the runtime's own source ARN.

### Why this, not the existing optimization loop

AgentCore's optimization workflow (`optimization/optimize_agent.py` in this sample) compares
two variants concurrently (A/B) or against a validation batch. That's the wrong tool for
"has this same configuration gotten worse over time," since both arms of an A/B test degrade
together and the comparison reports nothing. This feature tracks one configuration against
its own persisted history instead. Full reasoning is in the feature's own README.

### User experience

**Before:** No signal exists when the agent's live quality degrades (e.g. after a model
version update with no corresponding deploy). Degradation is only found via manual poking
around Evaluations dashboards or when a user complains.

**After:** A CloudWatch alarm fires per evaluator, naming which score stream drifted, with a
dashboard showing pressure toward the alarm threshold, samples seen, and baseline mean per
evaluator.

### Scope

Self-contained under `market-trends-agent/drift_detection/`. Does not modify the agent's
runtime behavior except for two demo-only drift triggers gated behind env vars
(`STALE_PRICES`, `SKIP_PROFILE_STEP`) that are no-ops unless explicitly set.

Contributor guide

Open the contributing guide

Research direction

Start with the Market Trends Agent directory and the feature's README, then read optimization/optimize_agent.py to understand the existing evaluation workflow. Review the proposed drift_detection/ tools, including watch.py, shape_report.py, and induce_drift.py. Done means scheduled detection, per-evaluator CloudWatch alarms and dashboard reporting, deploy/teardown scripts, and the stated IAM and trust-policy fixes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai, cli, cloud, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.