awslabs / awslabs/agent-evaluation
[Feature Request] Guardrail option for Bedrock KB target (maybe SageMaker too?)
- Dominant language
- Python
- Stars
- 372
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
[Amazon Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html) can be applied in a range of ways:
1. ✅ In Bedrock Agents, a guardrail is configured as part of the agent itself, so `agent-evaluation` already supports testing BR agent targets with guardrail in the loop
2. ❌ In Bedrock KBs, the guardrail ID & version must be specified when calling [RetrieveAndGenerate](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_RetrieveAndGenerate.html), and `agent-evaluation` doesn't yet support configuring this
3. ❓If there's interest, maybe it could be interesting to expand the SageMaker Endpoint target to offer an option to use an Amazon Bedrock Guardrail via the [ApplyGuardrail API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html)?
Of course it's possible to achieve already by building a custom target, and (3) seems potentially a bit niche - but I manage a [workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/ab6c96d3-53cf-4730-b0fe-f4762dbbb6eb) that demonstrates agent-evaluation on a KB for which feature (2) would be helpful!
Contributor guide
Research direction
Start by tracing the existing Bedrock agent target and its RetrieveAndGenerate integration; the issue names no repository files or tests. Define the Bedrock KB option around guardrail ID and version, then verify the target can send those values and add coverage; clarify separately whether SageMaker and ApplyGuardrail are in scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ai, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100