AgentThreatBench: Benchmark to evaluate guardrail effectiveness against OWASP agentic threats (ASI01/ASI06)

Proposal: Use AgentThreatBench to benchmark NeMo Guardrails effectiveness

AgentThreatBench is an evaluation suite that operationalizes the OWASP Top 10 for Agentic Applications (2026) into executable benchmark tasks. It was recently merged into the official UK AI Safety Institute's inspect_evals repository.

Why this is directly relevant to NeMo Guardrails

NeMo Guardrails is designed to prevent exactly the threats that AgentThreatBench measures:

AgentThreatBench could serve as a standardized test suite for measuring how effectively NeMo Guardrails configurations prevent these attacks — providing a before/after comparison that demonstrates guardrail value.

Example integration

from nemoguardrails import RailsConfig, LLMRails
from inspect_evals.agent_threat_bench import run_benchmark

# Run baseline (no guardrails)
baseline_results = run_benchmark(model="openai/gpt-4o")

# Run with NeMo Guardrails
config = RailsConfig.from_path("./guardrails_config")
rails = LLMRails(config)
guarded_results = run_benchmark(model=rails)

# Compare security scores
print(f"Baseline security score: {baseline_results.security_score}")
print(f"Guarded security score: {guarded_results.security_score}")

Resources

• Benchmark docs: https://ukgovernmentbeis.github.io/inspect_evals/evals/safeguards/agent_threat_bench/
• Source code: https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/agent_threat_bench
• OWASP standard: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

Would love to discuss adding AgentThreatBench to the NeMo Guardrails test suite or documentation as a reference benchmark for agentic security evaluation.

Start by reviewing the inspect_evals agent_threat_bench source and benchmark documentation linked in the issue, then inspect the NeMo Guardrails test suite and the RailsConfig/LLMRails entry points. The issue does not choose between a test-suite integration and a documentation reference, so define that scope first; completion should include a reproducible baseline-versus-guarded evaluation for the stated threats.