awslabs / awslabs/agent-evaluation

[Feature Request] Guardrail option for Bedrock KB target (maybe SageMaker too?)

Open
#83 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
372
Forks
51
PR merge metrics
No merged PRs in 30d

Description

[Amazon Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html) can be applied in a range of ways:

1. ✅ In Bedrock Agents, a guardrail is configured as part of the agent itself, so `agent-evaluation` already supports testing BR agent targets with guardrail in the loop
2. ❌ In Bedrock KBs, the guardrail ID & version must be specified when calling [RetrieveAndGenerate](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_RetrieveAndGenerate.html), and `agent-evaluation` doesn't yet support configuring this
3. ❓If there's interest, maybe it could be interesting to expand the SageMaker Endpoint target to offer an option to use an Amazon Bedrock Guardrail via the [ApplyGuardrail API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html)?

Of course it's possible to achieve already by building a custom target, and (3) seems potentially a bit niche - but I manage a [workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/ab6c96d3-53cf-4730-b0fe-f4762dbbb6eb) that demonstrates agent-evaluation on a KB for which feature (2) would be helpful!

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing Bedrock agent target and its RetrieveAndGenerate integration; the issue names no repository files or tests. Define the Bedrock KB option around guardrail ID and version, then verify the target can send those values and add coverage; clarify separately whether SageMaker and ApplyGuardrail are in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.