awslabs / awslabs/agentcore-samples
Add Agent Skills evaluation sample
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
## Description
Add a focused sample under `01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/skills-evaluation/` for the two built-in AgentCore skill evaluators:
- `Builtin.SkillSelectionAccuracy`
- `Builtin.SkillInstructionFollowing`
## Proposed sample
- Reuse the existing shared HR Assistant with opt-in Strands `AgentSkills` support.
- Include simple PTO-planning and benefits-advisor `SKILL.md` examples.
- Demonstrate on-demand evaluation using session OpenTelemetry records and the `Evaluate` API.
- Include a no-skill control to explain documented skip behavior.
- Document deployment, expected results, troubleshooting, and cleanup.
## Motivation
The AgentCore developer guide documents both skill evaluators, but this repository does not currently include an end-to-end runnable sample showing skill loading, trace collection, evaluator calls, and one-result-per-skill-invocation behavior.
Documentation: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/skill-evaluators.html
Contributor guide
Research direction
Start in 01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/skills-evaluation/ and inspect the existing shared HR Assistant and its AgentSkills support. Read the linked skill-evaluator documentation, then trace how session OpenTelemetry records feed the Evaluate API. Done means a runnable sample with PTO and benefits SKILL.md files, a no-skill control, expected results, troubleshooting, deployment, and cleanup guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ai, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100