awslabs / awslabs/agentcore-samples
Add evaluation samples for newly-supported frameworks (OpenAI Agents SDK, LlamaIndex)
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
## Summary
Amazon Bedrock AgentCore Evaluations recently extended support beyond Strands and LangGraph to additional agent frameworks, including **OpenAI Agents SDK** and **LlamaIndex** (see [Supported agent frameworks](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/supported-frameworks.html)).
Today `01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/` only demonstrates evaluation for the built-in Strands HR Assistant. There is no runnable sample showing how to make an agent built with one of the newly-supported frameworks evaluable end-to-end.
## Proposal
Add a `supported-frameworks/` subfolder under `02-evaluate/` with one self-contained code sample per newly-supported framework, starting with:
- **`openai-agents/`** — the shared HR Assistant re-implemented with the OpenAI Agents SDK, using OpenAI GPT-5.5 on Amazon Bedrock (OpenAI-compatible Responses API + Bedrock API key).
- **`llamaindex/`** — the same HR Assistant as a LlamaIndex `FunctionAgent` workflow, using Amazon Nova Lite via `BedrockConverse`.
Each sample deploys to AgentCore Runtime with OpenTelemetry instrumentation (auto-discovered by ADOT), persists conversation history in AgentCore Memory, and evaluates the agent with built-in + custom LLM-as-a-judge evaluators (on-demand and online), plus an AgentCore CLI re-evaluation example. Reusing the same HR Assistant domain keeps ground truth consistent so results are comparable across frameworks.
## Value
Gives users a copy-runnable, framework-specific starting point for AgentCore Evaluations, and a generalized recipe ("make any framework agent evaluable") they can apply to the other supported frameworks.
This supports the AgentCore Evaluations launch content for the newly-supported frameworks.
Contributor guide
Research direction
Start by reading the existing samples under 01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/ and the current Strands HR Assistant implementation. Add supported-frameworks/openai-agents/ and supported-frameworks/llamaindex/ samples that deploy to AgentCore Runtime, persist conversation history, and demonstrate on-demand and online evaluations plus CLI re-evaluation. Done means both samples are runnable and use comparable HR Assistant ground truth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ai, backend, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100