awslabs / awslabs/agentcore-samples

Add evaluation samples for newly-supported frameworks (OpenAI Agents SDK, LlamaIndex)

Open
#1,773 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
30

Description

## Summary

Amazon Bedrock AgentCore Evaluations recently extended support beyond Strands and LangGraph to additional agent frameworks, including **OpenAI Agents SDK** and **LlamaIndex** (see [Supported agent frameworks](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/supported-frameworks.html)).

Today `01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/` only demonstrates evaluation for the built-in Strands HR Assistant. There is no runnable sample showing how to make an agent built with one of the newly-supported frameworks evaluable end-to-end.

## Proposal

Add a `supported-frameworks/` subfolder under `02-evaluate/` with one self-contained code sample per newly-supported framework, starting with:

- **`openai-agents/`** — the shared HR Assistant re-implemented with the OpenAI Agents SDK, using OpenAI GPT-5.5 on Amazon Bedrock (OpenAI-compatible Responses API + Bedrock API key).
- **`llamaindex/`** — the same HR Assistant as a LlamaIndex `FunctionAgent` workflow, using Amazon Nova Lite via `BedrockConverse`.

Each sample deploys to AgentCore Runtime with OpenTelemetry instrumentation (auto-discovered by ADOT), persists conversation history in AgentCore Memory, and evaluates the agent with built-in + custom LLM-as-a-judge evaluators (on-demand and online), plus an AgentCore CLI re-evaluation example. Reusing the same HR Assistant domain keeps ground truth consistent so results are comparable across frameworks.

## Value

Gives users a copy-runnable, framework-specific starting point for AgentCore Evaluations, and a generalized recipe ("make any framework agent evaluable") they can apply to the other supported frameworks.

This supports the AgentCore Evaluations launch content for the newly-supported frameworks.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing samples under 01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/ and the current Strands HR Assistant implementation. Add supported-frameworks/openai-agents/ and supported-frameworks/llamaindex/ samples that deploy to AgentCore Runtime, persist conversation history, and demonstrate on-demand and online evaluations plus CLI re-evaluation. Done means both samples are runnable and use comparable HR Assistant ground truth.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai, backend, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.