awslabs / awslabs/agent-evaluation

Add Target for AgentCore Runtime

Open
#113 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
372
Forks
51
PR merge metrics
No merged PRs in 30d

Description

Today I had a quick go at sketching out a `Target` implementation for agents deployed on Bedrock AgentCore Runtime.

It seems generally pretty feasible within the current patterns... But I think the main design decision/friction point will be how to handle the facts that AgentCore supports arbitrary request & response payloads, and expects response streaming?

In `SageMakerEndpointTarget`, this was [handled](https://github.com/awslabs/agent-evaluation/blob/5d2784af28bae4bc3a0edd651ce78def310faea1/src/agenteval/targets/sagemaker_endpoint/target.py#L33) by configuring:
- A template request payload JSON
- A JSONPath to overwrite in the request payload with the new prompt
- A JSONPath to query from the response (also assumed JSON) to extract the reply text.

The AgentCore docs propose a generic code sample [here](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html) for handling both `application/json` and `text/event-stream` response streams.

As far as I'm aware, agent-evaluation doesn't have a general solution for introducing multi-modal tests yet?

So, any opinions on how to set up configuration to make an AgentCore target suitably flexible but easy-to-use?

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked SageMakerEndpointTarget implementation and the AgentCore runtime invocation example. Compare how request templates, JSONPath extraction, application/json, and text/event-stream responses could be configured. Done means the project has an agreed, suitably flexible AgentCore target design, including how arbitrary payloads and streaming responses are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai, api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.