RFC: Synthetic evaluation dataset generation from real agent traces

## Executive summary

This SDK already stores and analyses **real** agent traces. The highest-value thing to do with that corpus, beyond measuring it, is to **derive evaluation datasets from it**.

Proposal: a synthetic-data module that turns real BQAA traces into evaluation cases — task inputs, expected trajectories, expected answers, and deliberately-constructed near-misses — so that eval suites are grounded in the distribution an agent actually sees in production rather than in what a prompt author imagines it sees.

Most synthetic eval generation today starts from a schema or a hand-written prompt and invents plausible-looking tasks. Starting from traces is a materially different position: real user phrasing, real tool-call sequences, real context sizes, real failure modes, and real class imbalance are all already in the data. That is the differentiator worth building around.

## What would be generated

1. **Eval cases from successful traces.** `(input, expected_trajectory, expected_answer)` triples derived from traces that completed correctly. Feeds trajectory matching directly.
2. **Perturbations and counterfactuals.** Mutate a known-good trace to produce near-misses — wrong tool argument, dropped step, reordered calls, truncated context. These test whether a grader *discriminates*, which pass/fail cases alone do not.
3. **Rare-failure oversampling.** Genuine failure modes are rare in production traces. Amplifying them is the standard fix for the class-imbalance problem that otherwise makes a suite look healthy while testing almost nothing interesting.
4. **Shareable corpora.** Regenerate traces that preserve structure and statistics without reproducing customer content, so eval fixtures can be shared or published.

## Where this plugs into the existing SDK

This is deliberately mostly composition of things that already exist, not a new engine:

| Existing module | Role |
|---|---|
| `eval_suite.py` | Generated cases become suites; graduation and saturation detection already handle lifecycle |
| `evaluators.py` (trajectory matching) | Expected trajectories come from real ones rather than being hand-authored |
| `categorical_evaluator.py` | Labels the seed traces so generation can stratify by category |
| `multi_trial.py` (pass@k / pass^k) | Decides whether a generated case is actually discriminative or just noise |
| `eval_validator.py` | Already validates for ambiguous tasks, class imbalance and suspicious thresholds — this is exactly the guardrail synthetic generation needs, and it exists today |
| `insights.py` | Facet extraction to stratify seeds so generation covers the real distribution instead of over-sampling the easy middle |
| `AI.GENERATE` / `AI.EMBED` / `AI.CLASSIFY` | Generation and near-duplicate detection in-database |

`eval_validator.py` is worth calling out specifically: a synthetic generator without a quality gate produces confident garbage, and the gate is already written.

## Proposed plan

**Phase 0 — Seed selection and stratification.** Given a trace table, select and stratify seed traces by outcome and facet. No generation yet. Deliverable: a documented seed-selection API plus a report of what the seed distribution actually looks like. This is independently useful and will probably reveal that some failure classes are too rare to sample at all.

**Phase 1 — Case extraction (no synthesis).** Convert real traces directly into eval cases. Strictly speaking this is extraction, not generation, but it is the correct first milestone: it establishes the case schema, wires trajectory matching, and produces a working suite before any model is involved.

**Phase 2 — Perturbation.** Generate near-miss variants from known-good traces. Rule-based mutations first (drop a step, corrupt an argument, reorder), since they are deterministic and cheap to validate, before anything LLM-driven.

**Phase 3 — LLM synthesis.** Generate genuinely novel cases conditioned on seed traces, gated by `eval_validator` and a near-duplicate check against the seed corpus.

**Phase 4 — Correlation study.** Measure whether scores on the synthetic suite predict scores on a held-out real suite. Without this, the whole feature is unfalsifiable.

Phases 0–2 need no new generation capability and should be shipped before Phase 3 is scoped.

## Risks and open questions

1. **Leakage is the central risk.** Cases derived from traces an agent has already produced will inflate its own scores. Holdout discipline has to be designed in from Phase 0, not bolted on — including whether a case derived from agent A's trace may be used to evaluate agent A at all.
2. **Privacy.** Traces contain customer content. Anything intended to be shareable needs an explicit near-duplicate rejection step against the seed corpus, and "it was generated by a model" is not by itself evidence that source content was not reproduced.
3. **Validity ceiling.** An expected answer extracted from a real trace is only as correct as that trace was. Deriving ground truth from unverified production behaviour risks encoding current bugs as the specification.
4. **Does it correlate?** Whether synthetic eval performance predicts real performance is an empirical question. It should be measured (Phase 4) rather than assumed, and the feature should be considered unproven until it is.
5. **Scope boundary.** This proposal is deliberately scoped to the SDK consumption layer over traces, using generally available BigQuery AI functions. Any deeper engine-level generation capability is out of scope here and would be tracked separately.

## Related

- #435 — AgentForensics failure taxonomy. Directly complementary: the taxonomy supplies the label space for rare-failure oversampling in item 3 above.
- #428 — golden Q&A matching (the producer for expected answers)
- #430 — canonical evaluation rubrics
- #429 — turn tagging and `sub_trajectories`

Prior art worth reviewing: Databricks and MLflow both ship synthetic data generation aimed at agent evaluation, so there is public precedent for the shape of this and for what users expect from it.

---

Opening for discussion. The specific thing I would like feedback on is whether Phase 1 (extraction) is worth shipping on its own, or whether it is too thin to be useful without perturbation.

Start by reviewing eval_suite.py, evaluators.py, categorical_evaluator.py, multi_trial.py, eval_validator.py, and insights.py to understand the existing evaluation and trace APIs. The issue is an RFC with open scope, leakage, privacy, and validity questions, so a maintainer-approved Phase 0 or Phase 1 design is needed before implementation can begin. Done would be a concrete, agreed API and milestone rather than the full multi-phase proposal.

Written by the indexing model from the issue text.