RFC: Port the evidence-led agent red-team benchmark into AGT's testing suite

## Proposal

Should AGT add a completed, evidence-led benchmark to its existing red-team testing suite?

The reference implementation is already built and running in [kerberosmansour/AGT-Embeddings-Experiment](https://github.com/kerberosmansour/AGT-Embeddings-Experiment). Its latest completed release gate joined **54,034 static detector records** with **250 live, sandboxed behavioural records**. It also includes a **24-scenario CI smoke suite** and a balanced **240-scenario measurement suite**. The capability is documented in [issue #41](https://github.com/kerberosmansour/AGT-Embeddings-Experiment/issues/41) and the completed consolidation is in [PR #40](https://github.com/kerberosmansour/AGT-Embeddings-Experiment/pull/40).

AGT already has useful red-team coverage. This proposal would add to it, not replace it. The main difference is that the benchmark follows an attack beyond the initial verdict: it records what the agent attempted, whether the action executed, and which control stopped it.

### At a glance

| Area | AGT today | Completed reference benchmark |
|---|---|---|
| Prompt defences | 17 static defence vectors | Uses detector results as one evidence source, then joins them to action outcomes |
| Adversarial playbooks | 5 built-in playbooks containing 13 attack steps | 24-scenario smoke suite plus 240-scenario balanced measurement suite |
| Policy regression | 28 scenarios run against 3 starter packs (84 policy checks) | Six cross-layer trap classes, including attack, evasion, hard-benign, and near-miss cases |
| Corpus-scale measurement | Existing suites are focused on targeted regression and resilience testing | 54,034 labelled static records in the completed release run |
| Agent behaviour | Playbook resilience and policy decisions | Deterministic mock traces plus 250 live sandboxed records in the completed release run |
| Reporting | Grades, pass rates, resilience scores, and policy outcomes | Detection-to-action matrix, containment, attack success, false positives, and utility preservation |

These numbers are not a head-to-head score: the suites do different jobs. AGT's current tests provide focused product regression coverage. The reference benchmark adds broader measurement, explicit benign controls, and action-level evidence.

### Feedback requested

Would maintainers and contributors find this useful as part of AGT? In particular:

- Should the completed capability be ported into AGT's testing suite?
- Is `tests/redteam` plus `agent-sre/chaos` the right ownership boundary?
- Should an initial contribution include only the deterministic smoke and mock-behavioural tiers, with the larger corpus and live tier following separately?
- Which of the six trap classes would be most valuable to AGT users?

## Why this may be useful

An attack being detected does not necessarily mean it was contained. A tool action can still execute after a warning, while a downstream control can sometimes stop an attack that the detector missed. The benchmark makes both cases visible by recording the detection verdict and action result for the same scenario:

| | Action blocked / contained | Action executed |
|---|---|---|
| **Detected** | Expected result | High-severity control failure |
| **Undetected** | Defence in depth held | Worst-case outcome |

This gives maintainers a clearer view of where a control worked, where it failed, and what should be fixed next. It also checks whether benign work still succeeds, so blocking everything cannot produce a strong result.

### Threat coverage: six trap classes

| Trap class | Attack surface | Examples |
|---|---|---|
| Content Injection | Content the agent reads | Hidden HTML comments, poisoned tool output, human-sees-X / agent-reads-Y format tricks |
| Semantic Manipulation | Meaning of the task | Authority spoofing, roleplay legitimacy, goal reframing, making unsafe requests sound routine |
| Cognitive State | Memory and session state | Poisoned memory, cross-session echoes, preference manipulation, stale context |
| Behavioural Control | Tool use and actions | Approval fatigue, tool-schema confusion, irreversible-pressure prompts, least-privilege blur |
| Systemic | The wider agent ecosystem | Cross-agent relay, MCP registry misdirection, package-name confusion, delegation loops |
| Human-in-the-Loop | Approval and review workflows | Fake approvals, social proof and time pressure, visible-safe / hidden-unsafe requests |

## Detailed Design

The reference implementation provides the following capabilities for adaptation:

1. **Typed scenario, result, and tool-trace contracts**
- Each scenario identifies its trap class, target layer, delivery surface, expected behaviour, and evidence level.
- Action traces distinguish `attempted`, `executed`, and `blocked_at` rather than inferring containment from model text.

2. **Layered execution tiers**

| Tier | Size | Purpose |
|---|---:|---|
| Smoke suite | 24 scenarios | Fast, deterministic CI regression check |
| Measurement suite | 240 scenarios | Per trap class: 8 canonical attacks, 16 evasions, 8 hard-benign cases, and 8 near misses |
| Static corpus tier | 54,034 records in the completed run | Corpus-scale detector evidence |
| Live behavioural tier | 250 pre-registered records plus a benign utility arm in the completed run | Real agent behaviour inside a Linux sandbox; opt-in |

3. **Explicit evidence levels**
- Declared evidence.
- Static detector evidence.
- Deterministic mock behavioural evidence.
- Live sandboxed behavioural evidence.

Reports cannot promote a result above the level at which it was produced.

4. **Joint reporting**
- Detection verdicts are joined to action outcomes for the same scenario.
- Reports include canonical catch rate, evasion degradation, false-positive rate, utility preservation, action-level attack success rate, containment rate, and benign false-block rate.
- The off-diagonal `undetected → contained` and `detected → executed` cases remain visible rather than being hidden by a single aggregate score.

5. **Safety and integrity controls**
- Generated reports are metadata-only and are checked for raw payloads, secrets, live URLs, and personal data.
- Live tests fail closed when the required sandbox cannot be established; they do not silently fall back to weaker evidence.
- Live execution denies network egress, scrubs the environment, and prevents access to the host filesystem.
- Reports are explicitly non-certifying and carry reproducibility hashes for the scenario set, corpus, and artifacts.

### Proposed AGT integration

A staged port could preserve AGT's existing interfaces and package boundaries:

1. Add the deterministic schemas, fixtures, mock harness, and smoke runner under `tests/redteam/`.
2. Place reusable adversarial execution support under `agent-sre/chaos`, consistent with the completed package-boundary decision in [AGT issue #200](https://github.com/microsoft/agent-governance-toolkit/issues/200).
3. Optionally expose the benchmark through `agt red-team` once the repository-level contract is stable.
4. Keep live behavioural runs opt-in and Linux-only initially; the default CI path remains deterministic and side-effect-free.
5. Treat external corpora as pinned inputs with manifests rather than committing large or sensitive payload collections by default.

The first contribution could be limited to schemas, the 24-scenario smoke suite, mock behavioural traces, joint reporting, and artifact hygiene. The full measurement and live tiers could follow after maintainers validate placement and maintenance expectations.

## Alternatives Considered

1. **Keep the benchmark only in the external repository.** This avoids new AGT maintenance cost but fragments the testing story and prevents AGT's own controls from using the richer action-level evidence in CI.
2. **Port only the additional scenarios.** This increases attack coverage but loses the central distinction between detection, attempted action, containment, and execution.
3. **Extend only the existing resilience score.** A single score is simpler, but it can hide the high-severity `detected → executed` case and benign utility failures.
4. **Import the complete corpus into AGT.** This is easy to reproduce locally but increases repository size and payload-handling risk. Pinned external inputs plus manifests are a safer default.

## Security Implications

This proposal expands security testing without changing AGT's runtime trust model. The main risks are test payload leakage, unintended live side effects, false confidence from weak evidence, and test-set tuning.

The proposed controls are:

- Metadata-only, raw-free report artifacts with a fail-closed hygiene check.
- No secrets, personal data, or live targets in committed scenarios or generated reports.
- OS-enforced isolation for live behavioural tests, including denied network egress, scrubbed environment variables, and no host filesystem access.
- Refusal to claim live evidence when the sandbox or required execution path is unavailable.
- Pre-registered live samples and frozen thresholds before results are reviewed.
- A permanent non-certification statement in machine-readable and human-readable reports.

## Migration / Backward Compatibility

The port can be additive:

- Existing `agt red-team` commands and playbooks remain unchanged.
- Existing `tests/redteam` cases continue to run.
- The deterministic smoke tier can be introduced as a separate CI job.
- Live testing remains opt-in and does not affect contributors without the required Linux sandbox.
- Public CLI changes, if desired, can follow as a separate reviewed change after the test contracts stabilize.

## Scope

Cross-package (2-3 packages): repository-level tests, `agent-sre`, and optionally `agent-compliance` for CLI exposure.

## Target Placement

Existing `tests/redteam` for the benchmark contract and CI suite, with reusable adversarial execution support in `agent-sre/chaos`. Optional public CLI integration can remain a later decision.

## Prior Art

- Completed reference implementation: [kerberosmansour/AGT-Embeddings-Experiment](https://github.com/kerberosmansour/AGT-Embeddings-Experiment)
- Capability overview and usage: [kerberosmansour/AGT-Embeddings-Experiment#41](https://github.com/kerberosmansour/AGT-Embeddings-Experiment/issues/41)
- Completed implementation: [kerberosmansour/AGT-Embeddings-Experiment#40](https://github.com/kerberosmansour/AGT-Embeddings-Experiment/pull/40)
- Existing AGT red-team guide: [`tests/redteam/README.md`](https://github.com/microsoft/agent-governance-toolkit/blob/main/tests/redteam/README.md)
- Existing AGT red-team CLI tutorial: [`docs/tutorials/47-red-team-testing.md`](https://github.com/microsoft/agent-governance-toolkit/blob/main/docs/tutorials/47-red-team-testing.md)
- Prior AGT package-boundary decision: [microsoft/agent-governance-toolkit#200](https://github.com/microsoft/agent-governance-toolkit/issues/200)

## Checklist

- [x] I have searched existing issues and RFCs for duplicates.
- [x] I have read the ADR index for related decisions.
- [x] I am willing to implement this RFC or help review an implementation.

Start with tests/redteam/README.md and docs/tutorials/47-red-team-testing.md, then read the package-boundary decision in issue #200. Review the reference implementation before scoping work across tests/redteam and agent-sre/chaos. Done means an agreed staged port with deterministic schemas, the 24-scenario smoke suite, mock traces, joint reporting, and artifact-hygiene checks.

Written by the indexing model from the issue text.