awslabs / awslabs/agentcore-samples

Add Agent Skills evaluation sample

Open
#1,970 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
30

Description

## Description

Add a focused sample under `01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/skills-evaluation/` for the two built-in AgentCore skill evaluators:

- `Builtin.SkillSelectionAccuracy`
- `Builtin.SkillInstructionFollowing`

## Proposed sample

- Reuse the existing shared HR Assistant with opt-in Strands `AgentSkills` support.
- Include simple PTO-planning and benefits-advisor `SKILL.md` examples.
- Demonstrate on-demand evaluation using session OpenTelemetry records and the `Evaluate` API.
- Include a no-skill control to explain documented skip behavior.
- Document deployment, expected results, troubleshooting, and cleanup.

## Motivation

The AgentCore developer guide documents both skill evaluators, but this repository does not currently include an end-to-end runnable sample showing skill loading, trace collection, evaluator calls, and one-result-per-skill-invocation behavior.

Documentation: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/skill-evaluators.html

Contributor guide

Open the contributing guide

Research direction

Start in 01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/skills-evaluation/ and inspect the existing shared HR Assistant and its AgentSkills support. Read the linked skill-evaluator documentation, then trace how session OpenTelemetry records feed the Evaluate API. Done means a runnable sample with PTO and benefits SKILL.md files, a no-skill control, expected results, troubleshooting, deployment, and cleanup guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.