RFC: AgentForensics collaboration — automated failure taxonomy and span-level trace auditing over BQAA traces

## Executive summary

**AgentForensics** proposes a diagnostic framework that inspects agent execution traces on existing benchmarks (KramaBench, LakeQA, Spider 2.0) to automatically identify and categorize the exact failure point in a run.

That is a close fit for this SDK, and the overlap is complementary rather than duplicative. BQAA already provides the substrate AgentForensics would otherwise have to build — trace capture into BigQuery, trace/DAG reconstruction, an LLM classification engine with strict label validation, and BigQuery-scale aggregation. What AgentForensics brings is the part the SDK does not have: a **literature-grounded failure taxonomy**, **span-level failure localization**, and a **labeled corpus from public benchmarks** to validate against.

The short version: AgentForensics supplies the taxonomy and the ground truth; BQAA supplies the storage, reconstruction and classification engine. Neither side has to build the other's half.

## What the SDK already provides

Relevant primitives that exist today:

| Capability | Module | Relevance to AgentForensics |
|---|---|---|
| Trace reconstruction + DAG visualization | trace/observability layer | Recovering the step sequence a failure sits in |
| Categorical evaluation into **user-defined** categories via `AI.GENERATE`, with strict category validation | `categorical_evaluator.py` | This is a taxonomy classifier already — it just needs a taxonomy |
| Facet extraction, aggregation, top-N distributions across sessions | `insights.py` | Per-benchmark failure-mode distributions |
| Trajectory matching (exact / in-order / any-order) | `evaluators.py` | Detecting where an actual trajectory diverges from a reference |
| Multi-trial evaluation with pass@k / pass^k | `multi_trial.py` | Separating deterministic failures from flaky ones |
| Agent Context Graph | `context_graph.py` | Decision lineage around the failure point |
| Eval suite lifecycle, graduation, saturation detection | `eval_suite.py` | Turning a labeled failure set into a durable regression suite |

## What is genuinely missing (the collaboration surface)

1. **No canonical failure taxonomy.** `categorical_evaluator` classifies into whatever categories you define; there is no shipped, defensible set of agent failure modes.
2. **Classification is session-level, not span-level.** AgentForensics' core claim is locating *the exact failure point*. That requires attributing a category to a `span_id`, not a session.
3. **No public labeled trace corpus.** Every evaluator in this SDK is currently validated against synthetic or internal data.
4. **No benchmark harness.** Nothing here runs a public benchmark end-to-end and lands the traces.

## Proposed plan

**Phase 0 — Instrument the benchmarks.** Run the benchmark agents with BQAA telemetry enabled so traces land in BigQuery. Deliverable: a reproducible harness under `examples/` that executes a benchmark suite and produces a trace dataset. Open question below on which ingestion path applies.

**Phase 1 — Express the taxonomy as data, not code.** AgentForensics contributes the failure taxonomy; we encode it as `CategoricalMetricDefinition` / `CategoricalMetricCategory`. This is runnable on day one with no engine changes, which makes it a cheap way to test whether the taxonomy actually discriminates before anyone writes new code.

**Phase 2 — Span-level localization.** Extend categorical classification from session to span/turn granularity, emitting `(trace_id, span_id, failure_category, evidence, confidence)`. This is the one real engineering item and should reuse the turn-tagging and `sub_trajectories` work in #429 rather than inventing a parallel mechanism.

**Phase 3 — Validation and golden set.** Human-label a stratified sample; measure classifier agreement against it. The labeled set becomes a public regression fixture — which is independently valuable to this SDK and connects to #428 and #430.

**Phase 4 — Reporting.** Reuse the `insights.py` aggregation path for per-benchmark failure-mode distributions, plus a notebook and dashboard panel.

Phases 0–1 are deliberately sequenced to produce a falsifiable result early: if the taxonomy does not separate failure modes on real benchmark traces, that is worth discovering before building Phase 2.

## What each side gets

- **AgentForensics** avoids building trace storage, reconstruction, an LLM classification harness and an aggregation layer, and gets BigQuery-scale analysis for free.
- **BQAA** gets a validated failure taxonomy, a public labeled trace corpus, and external validation of `categorical_evaluator` on non-synthetic data.

## Open questions

1. **Which framework do the benchmark reference agents use?** If they are ADK-based, the ADK plugin path works directly. If not, ingestion needs an OpenTelemetry route, which changes Phase 0's scope materially. This is the biggest unknown.
2. **Is the trace fidelity sufficient for localization?** Span-level attribution needs tool-call and step boundaries in the emitted trace. Benchmark harnesses often log only final answers and intermediate SQL, which would make Phase 2 impossible without first improving instrumentation.
3. **One taxonomy or three?** Spider 2.0 (text-to-SQL) and data-to-insight pipeline benchmarks may not share failure modes. A single taxonomy is more useful; per-benchmark dialects may be more accurate. Worth resolving in Phase 1.
4. **Can the resulting traces and labels be published?** The corpus is a large part of the value, and that depends on each benchmark's licensing.
5. **Ground truth for "the exact failure point"** — do the benchmarks provide step-level references, or is the reference only a final answer? If the latter, localization is evaluated against human labels only, which caps the achievable rigor.

## Related issues

- #429 — turn tagging and `sub_trajectories` (the natural substrate for span-level localization)
- #430 — canonical evaluation rubrics in core
- #428 — golden Q&A matching

---

Opening this for discussion rather than as a committed plan. Feedback on the phasing — particularly whether Phase 2 should precede a taxonomy validation result — is welcome.