GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK
Native agent_events snapshot writer for AgentForensics (EvalBench-adapter exit ramp)
- Dominant language
- Python
- Stars
- 47
- Forks
- 21
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 33
Description
Parent: #435. This is the **next engineering slice after the Week 0 freeze evidence** (PRs #455–#462). #455 is **merged** (the recordable MVP e2e demo); #456–#462 are **open and agent-review-complete**, and **must not be merged as part of this work**.
## Why (team verdict, 2026-08-30)
Unanimous keep-as-adapter verdict (Claude / Codex / Kimi + Field Brief bot):
- EvalBench is a **snapshot format we read** (`configs` / `results` / `scores` by `job_id`) — not a runtime library (nothing EvalBench appears in `pyproject.toml`) and not the partner.
- The customer object is a **session in `agent_events`**, not an EvalBench scenario.
The consequence is an exit ramp, not a removal: BQAA should be able to produce its own snapshot contract directly from production ADK `agent_events`, with the EvalBench adapter remaining an optional on-ramp.
## Goal
A **native writer** that starts from production ADK `agent_events` rows and emits the same BQAA-owned contract the `evalbench-import` adapter already produces:
- a **pinned, immutable snapshot** (versioned BQAA-owned tables + manifest, as `EvalBenchRun.materialize` does today),
- the **`failed_sessions` view** (the per-dataset view whose body is `failed_sessions_sql` pinned to the latest successful import),
- **G1 taxonomy labels** — for the widget-stock silence session `7e352c34` that means `task/planning`, `finalization`, `tool blockers`,
with **no EvalBench source tables anywhere in that path**. The existing `evalbench-import` adapter (#97) **stays as an optional path — do not delete it**.
## Contracts the writer must honor (all frozen in the Week 0 evidence)
- **Identity.** EvalBench `results.id` / `eval_id` is the first eight characters of the ADK `session_id` (`7e352c34` from `7e352c34-4c1c-4395-acd5-fb3c8f215346`), falling back to the full id on collision (see the identity table in `examples/evalbench_mvp_e2e.md`). The native writer must preserve a **joinable identity** so the same session is the same object with or without the adapter.
- **Pin identity (native snapshot key).** Retain `(job_id, import_version)` — the same tuple `EvalBenchRun.materialize` already uses. Do **not** invent a different native key. Snapshot event rows, BQAA-owned score rows, and the manifest row all carry both fields. The `failed_sessions` view body is `failed_sessions_sql` with `@job_id` / `@import_version` rendered as literals for the latest successful **native** publication. Latest-success rule: the newest committed manifest row for that `job_id` (`ORDER BY imported_at DESC, import_version DESC LIMIT 1`). Manifest `generation_id` plus `view_policy` pin the view body the same way the adapter already does. For this slice the job is `mvp-e2e-real-traces`.
- **Read vs write.** **Read** production ADK `agent_events` only (`test-project-0728-467323.bqaa_e2e_real.agent_events` for the widget-stock acceptance). **Write** BQAA-owned snapshot events table + BQAA-owned scores table + manifest + the `failed_sessions` view. **Never write into production `agent_events`.** Never read EvalBench source `configs` / `results` / `scores` tables on this path.
- **Denominator.** `failed_sessions` is the denominator. A live/LLM judge is **not** the denominator. `returncode == 0` means *completed*, not *passed* — only the score policy decides *passed*.
- **Native score source and score policy.** Scores live in the **BQAA-owned snapshot scores table** (same schema the adapter already writes: `job_id`, `import_version`, `scenario_id`, `session_id`, `comparator`, `score`, `source_row`), **not** EvalBench source `scores` tables. Native score rows are derived **deterministically from the session** (fixture-testable; no live BQ and no live/LLM judge). The gate is existing `EvalScorePolicy`: `min_scores` (comparator → threshold) plus `missing_score_fails` (default **true**), persisted as canonical JSON on the manifest `view_policy` column (`_policy_pin`). `score_failed` is true iff any listed comparator fails: missing/`NULL` score when `missing_score_fails` is true, or `score < min_score`. Empty policy ⇒ `score_failed` is FALSE (`process_failed` / `missing_completion` still apply). For this slice's widget-stock session the frozen rubric (`examples/fixtures/week0_real_rubric.json`, `docs/week0_partner.md`) is comparator `goal_completion` with `min_score` 1.0 (`failed_session_score` 0 vs gold sibling `ab7535a5` score 1) and `missing_score_fails: true`, so `7e352c34` is `score_failed` and maps to `task/planning`.
- **G1 mapping** is already frozen in `src/bigquery_agent_analytics/failure_taxonomy.py` (`taxonomy_version: 0.1.0`, `g1_frozen: True`; see `docs/week0_g1_taxonomy.md`): `missing_completion` → `finalization`, `process_failed` → `tool blockers`, `score_failed` → `task/planning`. Freezing G1 did **not** start the clock.
- **Destination.** BQAA-owned snapshot tables + the `failed_sessions` view only. **Never write into production `agent_events`.**
- **D4 fail-closed.** The D4 consumer is **Hai-Yuan Cao only** — do not add D4 people as part of this slice.
- **Clock.** This issue does **not** start the six-week clock. The clock starts only when the first Week 1 snapshot job is kicked, which this issue does not do.
## Acceptance
- [ ] Widget-stock session `7e352c34-4c1c-4395-acd5-fb3c8f215346` from `test-project-0728-467323.bqaa_e2e_real.agent_events` can be snapshotted **without any EvalBench tables** and still yields `failed_sessions` + the G1 names (`task/planning`, `finalization`, `tool blockers`) + the punchline next-action (the agent never answered / never called `check_inventory`).
- [ ] The snapshot is keyed by `(job_id, import_version)` with `job_id=mvp-e2e-real-traces`; the `failed_sessions` view is pinned to the latest successful native publication of that tuple; score rows and `view_policy` use comparator `goal_completion` / `min_score` 1.0 / `missing_score_fails: true`.
- [ ] Offline fixture tests — unit tests require no live BigQuery.
- [ ] Nothing is written into production `agent_events`.
- [ ] The six-week clock is not started.
- [ ] D4 fail-closed remains Hai-Yuan Cao only.
## Out of scope
- Merging #456–#462.
- The preregistration seal.
- Clock start / kicking the Week 1 snapshot job.
- Dropping the EvalBench adapter (#97 stays optional).
- SANA fork.
- A live judge as the denominator.
## Related
- Parent: #435
- EvalBench adapter / import (stays optional): #97
- Week 0 evidence: #455 (merged MVP e2e demo), #461 (Week 0 freeze), #462 (post-freeze e2e)
- Demo (EvalBench adapter path, MVP e2e): https://caohy1988.github.io/evalbench/
Contributor guide
Research direction
Start by reading EvalBenchRun.materialize, src/bigquery_agent_analytics/failure_taxonomy.py, examples/fixtures/week0_real_rubric.json, and docs/week0_partner.md; use the existing adapter contract and EvalScorePolicy as reference. Exercise the offline fixture tests with the widget-stock session. Done means a native snapshot and pinned failed_sessions view use the BQAA-owned tables and (job_id, import_version), preserve the G1 labels, and never read or write production agent_events or EvalBench source tables.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100