Agent Quality Flywheel: BigQuerySampler over OTel/BQAA telemetry

# Agent Quality Flywheel: BigQuerySampler over OTel/BQAA telemetry

A governed agent-quality loop for ADK: load stable evaluation examples from BigQuery, run a candidate agent through ADK's evaluation path, compare it with the incumbent on the same immutable held-out cohort, and persist an auditable promotion decision. Automated OTel/BQAA failure mining feeds that loop after the sampler/evaluation bridge is proven.

📄 **Implementation companion:** [`docs/bigquery_sampler_eval_bridge_design.md`](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/blob/main/docs/bigquery_sampler_eval_bridge_design.md). This issue is the product and acceptance contract; the design doc must be updated to remain consistent with it before implementation.

## Product outcome

The first adopter is an ADK application or evaluation-platform team that already has a curated evaluation corpus or can materialize one in BigQuery.

The feature is successful when that team can:

1. bootstrap and validate the package-owned schemas and IAM;
2. run an existing ADK optimizer against a stable `optimizer_examples` snapshot;
3. evaluate candidate and incumbent on the **same immutable held-out IDs**;
4. receive a fail-closed, content-addressed decision receipt explaining why the candidate won, lost, or was ineligible for promotion; and
5. hand an approved digest to a deployment/canary system without this package auto-applying the change.

The strategic goal is not autonomous prompt mutation by itself. It is a trustworthy bridge from production evidence to a governed, reproducible candidate-versus-incumbent decision.

## Delivery boundary

### P0a — prove the sampler and trust contract

- `BigQuerySampler`, modeled on ADK's `LocalEvalSampler` (`perform_inference → evaluate`).
- A stable, curated `optimizer_examples` table populated from an existing eval/golden corpus or another approved content-bearing source. P0a must work without enabling production prompt logging.
- Candidate execution, deterministic score coverage, failure floors, candidate-versus-incumbent promotion gate, append-only attempt audit, and dry-run approval/canary handoff.
- One versioned, independently reference-grounded primary quality/correctness scorer. Deterministic observed execution, safety, and cost constraints may remain independent promotion floors or vetoes. General weighted multi-metric configuration and calibrated reference-free quality scoring are deferred until a demonstrated consumer requires them.

### P0b — automate telemetry mining

- SQL-driven candidate detection from deduplicated OTel-native tables/views and BQAA projections.
- `flywheel_candidates`, eligibility/curation, representative-cohort checks, telemetry trust controls, and scheduled materialization into `optimizer_examples`.
- Content-bearing telemetry is opt-in and policy-governed; default-privacy telemetry remains useful for detection and outcome signals but cannot by itself reconstruct replay inputs.

### Fast follow

- LangGraph patch-application/evaluation adapter.
- Sub-agent or tool-definition optimization.
- Live side-effectful tool replay.
- General weighted scoring recipes, calibration machinery for reference-free quality metrics, and additional evaluator types.
- Package-level `bq://` resolver if multiple source schemes justify the abstraction.
- Graduation/consolidation of all improvement demos after P0a identifies the canonical API.

## Non-negotiable trust invariants

1. **Same evidence:** candidate and incumbent are evaluated on the same immutable held-out ID set, replay fixtures, evaluator versions, model/config, and scoring policy.
2. **Complete coverage:** `set(result.scores) == set(requested_ids)`. Runtime failures receive the declared floor score and explicit failure metadata; missing or silently dropped IDs are impossible.
3. **Representative eligibility:** every run records the source population, eligible and excluded IDs/reasons, runnable coverage, and critical-slice distributions. Optimization and promotion fail closed below configured coverage or slice-representation thresholds.
4. **Grounding reaches the evaluator:** P0a's primary quality/correctness score is independently reference-grounded. The expected answer, authoritative tool result, explicit fixture outcome, or independently authored behavioral predicate must reach the evaluator and be recorded in the receipt. Uncalibrated reference-free quality metrics remain advisory; deterministic observed execution, safety, and cost constraints may still block promotion.
5. **No holdout leakage:** split by conversation/trace group and semantic-duplicate cluster, with a temporal cutoff. Candidate generation receives only training/validation evidence; the optimizer identity cannot read sealed promotion-holdout inputs or per-example outcomes. Detailed holdout results are audit-restricted, attempts per holdout generation are capped, and the holdout is retired or rotated when its configured adaptive-testing budget is exhausted.
6. **Stable identity:** source rows and examples use tenant/application/run-aware composite identity plus #316 native-row identity (`idempotency_key` / `source_position` when available). Repeated mining and retries are idempotent; reused session IDs never merge producers or runs.
7. **Observed evidence only:** event ordering, tool calls, outputs, and failures are captured from real execution or explicitly labeled fixtures. The system never infers or fabricates tool chronology from judge classifications.
8. **Immutable, serialized decision:** approval and canary handoff bind to a content-addressed manifest of the exact candidate, evaluated incumbent, corpus, evaluator, fixtures, config, and results. The handoff carries the candidate digest and expected incumbent digest or registry generation. The downstream promotion actor must atomically compare-and-set its canonical incumbent; a mismatch makes the attempt stale and requires re-evaluation.
9. **Fail closed:** incomplete cohorts, evaluator parse failures, unsafe replay contracts, identity collisions, schema-version mismatches, and uncalibrated promotion metrics block optimization or promotion rather than disappearing from the denominator.

## Data and ownership

| Surface | Role | P0 |
| --- | --- | --- |
| `optimizer_examples` | Stable, immutable runnable eval cases and split assignment | P0a |
| `flywheel_attempts` | Append-only record of every optimizer/candidate attempt and promotion decision | P0a |
| `flywheel_candidates` | Raw mined candidates, including ineligible/excluded rows and reasons | P0b |

This package owns DDLs/migrations, partitioning/clustering, retention, upgrade checks, IAM, and teardown for these tables. Raw OTel tables are provenance sources, not mining inputs: P0b reads #316/#317 deduplicated views (`*_dedup`, `agent_events_otlp`, or `bqaa_metrics`) and preserves exact native refs in provenance.

`optimizer_examples` contains only examples that passed the replay, policy, and identity eligibility checks. The full eligibility funnel and excluded rows remain in `flywheel_candidates` and the run manifest.

## Security and privacy posture

- Treat production prompts, tool outputs, labels, and telemetry-derived text as **untrusted optimizer input**.
- Replay uses isolated non-production credentials and fixtures/recorded results or explicitly approved read-only tool adapters. Network egress and side-effectful tools are denied by default; an unsafe replay contract makes an example ineligible.
- Content-bearing telemetry requires explicit opt-in, field allowlisting, secret/PII redaction before persistence/model use, tenant/application scoping, bounded retention, encryption/residency controls, and deletion propagation to derived examples where policy requires it.
- P0b records source trust tier and actor/tenant provenance; deduplicates and caps influence per actor/session/pattern; quarantines suspected prompt injection, poisoning, bot floods, and distribution anomalies; and keeps optimization and promotion populations isolated.
- Separate least-privilege identities for telemetry read, example/attempt write, optimizer execution, audit read, and approval. No cross-tenant joins by default.
- SQL-derived evaluators are parameterized, read-only, versioned, and restricted to approved views/templates with dataset allowlists, dry-run validation, timeout and maximum-bytes-billed limits. DDL/DML, scripting, remote functions, and unapproved external connections are denied.

## ADK compatibility

- Import `Sampler`, `UnstructuredSamplingResult`, and optimizer classes from their submodule paths; do not assume a flat `google.adk.optimization` export surface.
- The core SDK may retain its package-wide `google-adk>=1.0.0` floor, but the flywheel feature requires `google-adk>=1.31.1`. Its entry point capability-probes `google.adk.optimization.sampler` and returns an actionable unsupported-version error before BigQuery work or attempt creation.
- Pin and test an explicit flywheel matrix at the exact feature minimum and latest supported major. ADK 1.31.1 supplies the P0a sampler and prompt-optimizer surface; root-level `SkillToolset` optimization through `GEPARootAgentOptimizer` is capability-gated to ADK 2.4.0+ or fast-follow.
- No optimizer in the verified matrix modifies sub-agent prompts.

## Acceptance criteria

### Sampler and replay bridge

- [ ] `BigQuerySampler` is a `Sampler[UnstructuredSamplingResult]` modeled on `LocalEvalSampler` and honors the released `sample_and_score(candidate, example_set, batch, capture_full_eval_data)` contract for every supported ADK version.
- [ ] `get_train_example_ids()` and `get_validation_example_ids()` return IDs from an immutable run snapshot; only policy-approved, runnable examples may be requested.
- [ ] `result.scores` contains exactly one float for every requested ID. Run failures receive `on_run_failure` plus an ungated minimal `.data[id].run_failed` marker; heavy inputs/trajectories/tool outputs remain gated by `capture_full_eval_data`.
- [ ] Replay executes in the required sandbox with pinned tools, fixtures, auth policy, clock/randomness, timeout, and side-effect policy. Unsafe or incomplete replay contracts fail eligibility before optimizer invocation.

### Evaluation and promotion

- [ ] P0a supports one versioned, independently reference-grounded primary quality/correctness scorer and persists evaluator code/SQL hash, dependencies, inputs, raw per-example outcomes, and aggregation inputs.
- [ ] The expected answer, authoritative tool result, explicit fixture outcome, or independently authored behavioral predicate required for correctness is actually supplied to the evaluator; the receipt proves which evidence was used.
- [ ] Reference-free quality metrics are advisory in P0a and cannot affect promotion eligibility or ranking. Deterministic observed execution, safety, and cost constraints may be enforced as independent floors or vetoes without a subjective calibration program.
- [ ] Candidate and incumbent run on the same immutable held-out IDs and fixtures. Exact ID/count equality, complete score coverage, minimum paired improvement, critical-slice non-regression floors, and tie-keeps-incumbent behavior are enforced before approval handoff.
- [ ] The validation corpus includes representative successes, mined failures when available, safety/cost invariants, and critical slices. Eligibility coverage and representation thresholds are recorded and enforced.
- [ ] Train/validation/promotion splits are deterministic, group-aware, semantic-duplicate-aware, temporal, and immutable for the retention window.

### Audit and rollout

- [ ] Every optimizer and candidate attempt is written to `flywheel_attempts`, including rejected candidates, failed runs, exclusions, source refs, corpus snapshot, candidate/incumbent digests, evaluator/fixture/model/config versions, per-example results, approval state, and canary handoff state. It also records package-observed optimizer/candidate/evaluator call counts, provider-reported usage/tokens, elapsed time, BigQuery bytes processed/billed, usage-observability coverage, and — when available — estimated cost with price-catalog provenance.
- [ ] Each supported optimizer adapter declares whether its internal generation calls are observable and enforceable. Configurable per-attempt call/token/usage ceilings cover every instrumented model path; a policy requiring verified total usage rejects partially observable paths at preflight. Budget exhaustion cancels further observable calls, records `budget_exceeded`, makes the attempt ineligible, and assigns the declared failure floor plus metadata to every unprocessed requested ID whenever a sampler result is returned.
- [ ] Each attempt has a content-addressed run manifest that remains reproducible independently of mutable source tables and survives for the documented audit/rollback retention period.
- [ ] P0a never auto-applies a candidate. The wrapper may emit `approval_required` / `canary_requested`; a documented downstream actor/API owns `approved`, canary outcome, promotion, rollback, and kill-switch updates for the same approved digest.
- [ ] The handoff binds the approved candidate digest to the evaluated incumbent digest or registry generation. Immediately before promotion, the downstream actor atomically compare-and-sets its canonical incumbent from that expected value to the candidate; a mismatch records `stale_incumbent` in `flywheel_attempts` and requires fresh evaluation. A downstream system without this atomic operation receives an advisory, non-promotable handoff.
- [ ] Approval records approver identity, role separation, expiry, and the exact digest. The optimizer cannot approve its own output.
- [ ] The promotion-holdout generation, access policy, and attempt count are recorded in the receipt. Optimizer identities receive no detailed holdout inputs/outcomes, audit access is role-restricted, and the configured attempt cap forces holdout rotation or retirement.

### Telemetry mining (P0b)

- [ ] Candidate detection supports configured OTel-native deduplicated views and/or BQAA projections independently; it does not require both sources in one deployment.
- [ ] `flywheel_candidates` records the complete source population, source trust/identity, all eligibility outcomes and skip reasons, schema version, and exact telemetry provenance.
- [ ] Repeated windows/retries are idempotent and deterministic. Duplicate ingestion, reused session IDs, or overlapping projections cannot duplicate or merge candidates.
- [ ] Default-privacy telemetry is supported for detection/outcome signals. Replay inputs come from an independently approved corpus/resolver unless content logging was explicitly enabled and passed the content-governance policy.

### Operations and documentation

- [ ] Before implementation begins, the companion design is synchronized with the P0a/P0b boundary, deduplicated P0b inputs, all three owned tables, reference-grounded P0a scoring, trust invariants, usage limits, incumbent compare-and-set handoff, ADK matrix, and end-to-end sequence.
- [ ] Supported bootstrap creates and validates tables/views/jobs, partitioning/clustering, retention, IAM, budget limits, and schema compatibility; upgrade and teardown paths are documented and tested.
- [ ] Candidate/evaluator queries use timestamp partition bounds and maximum-bytes-billed limits; no run implicitly scans all historical partitions.
- [ ] The implementation composes with or explicitly supersedes existing evaluation/evolution surfaces (`quality_report.py`, `skill_evolution.py`, SDK evaluators, and the three improvement demos) so the SDK has one canonical promotion and audit contract.
- [ ] One end-to-end reference journey produces both an accepted and a rejected candidate, demonstrates a run failure without missing coverage, proves repeated-run idempotency, and reconstructs the final decision from persisted artifacts alone.
- [ ] Integration tests cover duplicate OTel delivery, two applications/users reusing a session ID, an excluded or failed held-out case, reference evidence not reaching the evaluator, poisoned/low-trust telemetry, budget exhaustion with complete score coverage, stale-incumbent promotion, adaptive holdout exhaustion/rotation, and approval-manifest tampering.

## Out of P0a

- Automated production-telemetry mining and curation (P0b).
- LangGraph patch application/evaluation.
- Sub-agent, tool-definition, or live side-effectful tool optimization.
- Generic weighted multi-metric recipe DSL.
- Automated production deployment or self-approval.

## Related work and dependencies

- #316 / #317: native OTel schemas, deduplicated views, native-row identity, and projections consumed by P0b.
- #351: score coverage, evaluator grounding, trace identity, chronology, and reproducible-report findings incorporated into the trust invariants above.
- #282: SDK cleanup and canonical public-surface work.
- Existing examples to consolidate after P0a: `agent_improvement_cycle/`, `self_evolving_agent_demo/`, and `skill_evolution_lab/`.

## Definition of done

P0a is done when a supported ADK optimizer can consume a stable BigQuery corpus, generate at least one candidate, compare candidate and incumbent on the same complete held-out snapshot, reject or advance it through the declared gate, and produce an immutable receipt that another engineer can reproduce without relying on mutable source telemetry. P0b begins only after that bridge, trust contract, and operational path have been demonstrated.

Start with docs/bigquery_sampler_eval_bridge_design.md and compare it with this issue's P0a acceptance criteria. Read ADK's LocalEvalSampler and the google.adk.optimization.sampler entry point before assessing the BigQuerySampler bridge. Done means the documented sampler, replay, evaluation, promotion, audit, and capability-gating contracts are implemented consistently.

Written by the indexing model from the issue text.