awslabs / awslabs/agent-evaluation
Add Target for AgentCore Runtime
- Dominant language
- Python
- Stars
- 372
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
Today I had a quick go at sketching out a `Target` implementation for agents deployed on Bedrock AgentCore Runtime.
It seems generally pretty feasible within the current patterns... But I think the main design decision/friction point will be how to handle the facts that AgentCore supports arbitrary request & response payloads, and expects response streaming?
In `SageMakerEndpointTarget`, this was [handled](https://github.com/awslabs/agent-evaluation/blob/5d2784af28bae4bc3a0edd651ce78def310faea1/src/agenteval/targets/sagemaker_endpoint/target.py#L33) by configuring:
- A template request payload JSON
- A JSONPath to overwrite in the request payload with the new prompt
- A JSONPath to query from the response (also assumed JSON) to extract the reply text.
The AgentCore docs propose a generic code sample [here](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html) for handling both `application/json` and `text/event-stream` response streams.
As far as I'm aware, agent-evaluation doesn't have a general solution for introducing multi-modal tests yet?
So, any opinions on how to set up configuration to make an AgentCore target suitably flexible but easy-to-use?
Contributor guide
Research direction
Start by reading the linked SageMakerEndpointTarget implementation and the AgentCore runtime invocation example. Compare how request templates, JSONPath extraction, application/json, and text/event-stream responses could be configured. Done means the project has an agreed, suitably flexible AgentCore target design, including how arbitrary payloads and streaming responses are handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ai, api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100