GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

Persist span-level G1 labels on the native snapshot (queryable localization)

Open
#469 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

Parent: #435 (AgentForensics RFC — automated failure taxonomy and span-level trace auditing). This is the next engineering slice after span-level G1 localization #466 (PR #467 library + PR #468 presenter e2e), and the native-writer analog of #463: publish already-landed localization as pinned snapshot rows.

## Sequencing

- #455 is merged. #456–#468 are open Week-0 freeze / native-writer / span-G1 evidence, agent-review-complete (or stacked on it). They must **not** be merged as part of this slice, and this slice must **not** wait for org APPROVE on them.
- The future PR must be stacked on #467 head `a7fd389` — not on #468's demo commits, not on #465, not on #462, and not on `main`. GitHub showing the #458–#467 file set against `main` is expected stacking, not a finding.
- G1 v0.1.0 was frozen in #461 (`failure_taxonomy.py`, `g1_frozen: true`).
- The native `agent_events` writer landed in #464 (head `f5c6109`): `NativeAgentEventsRun` publishes events / scores / manifest / `failed_sessions` from production ADK `agent_events` with no EvalBench source tables in the path.
- The span-level G1 library landed in #467 (head `a7fd389`, dual Codex+Kimi LGTM, P0=0 P1=0, CI green): `span_taxonomy.py` with `label_native_run` / `label_failed_session_spans` / `SpanFailureLabel.as_tuple()`.
- The span-G1 presenter e2e landed in #468 (head `0778125`, dual LGTM, P0=0 P1=0, CI green) — docs/script/tests only; no `src/`, no BigQuery write. #468 stays presenter-only.

## The gap

Phase 2 localization currently exists as a **library**, not as **snapshot data**. #467 already emits the RFC shape `(trace_id, span_id, failure_category, evidence, confidence)` for the widget-stock silence session — inspectable in Python tests and a `--fixture` presenter, but not part of the BQAA-owned snapshot contract. Session-level G1 is queryable via `failed_sessions`; span labels are not.

Do **not** reimplement `span_taxonomy` or invent a new taxonomy or a second localizer. The missing piece is *publication*: `NativeAgentEventsRun.materialize` (or a thin sibling that inherits it) should write a BQAA-owned span-labels table keyed by the frozen pin `(job_id, import_version)`, joinable to `failed_sessions` via the frozen `eval_id` rule.

Session-level `failed_sessions` + G1 remains the denominator. Span rows localize; they never classify or replace it.

## Goal

On a native snapshot of the widget-stock session `7e352c34` (full id `7e352c34-4c1c-4395-acd5-fb3c8f215346`), the published span-labels rows make the AGENT_STARTING silence inspectable as data:

- real `span_id` `b7ad6b7169203331`, `target_kind="gap_after_span"`;
- the frozen categories `task/planning`, `finalization`, `tool blockers`;
- evidence that no subsequent `TOOL_STARTING` / `check_inventory` / `AGENT_COMPLETED` occurred.

Source of truth is production ADK `agent_events` (native path). The EvalBench adapter (#97, `evalbench-import`) stays optional; this slice must not require EvalBench source `configs` / `results` / `scores`.

Reuse `label_native_run` with the frozen `EvalScorePolicy({"goal_completion": 1.0})` — per the #468 P1 finding, the default empty policy drops `task/planning`.

## Acceptance (widget-stock, fixture-testable, no live BigQuery)

- Session-level G1 for `7e352c34` still yields `task/planning`, `finalization`, `tool blockers` — unchanged.
- Published span-label rows for that session are exactly the three library rows (same span/trace ids, categories, `gap_after_span`, evidence strings) and carry `(job_id, import_version)` with `job_id=mvp-e2e-real-traces`.
- Identity: `eval_id = session_id[:8]` when unique in the snapshot / join set; full `session_id` on first-8 collision. Span rows must stay joinable to the same `eval_id` that `failed_sessions` / G1 already uses.
- No synthetic span identifiers. Rows without a real `span_id` fail closed.
- Frozen G1 names only (v0.1.0). No SANA fork, no new category strings. If a test proves a span cannot be labeled with the frozen three, preserve the failing case and defer the taxonomy change to a separate issue.
- Offline unit tests only: no live BigQuery, no `--synth`, no live/LLM judge, no production `agent_events` writes.
- Prefer inheriting `EvalBenchRun.materialize` / `NativeAgentEventsRun` rather than a parallel publisher. A thin CLI extension of `evalbench-native-import` is allowed; do not invent a new command family.

## Guardrails

- Clock OFF. This issue does not start the six-week clock, seal preregistration, or kick the Week 1 snapshot job.
- D4 fail-closed: Hai-Yuan Cao only.
- Never write production `agent_events`.
- Do not delete or rewrite `evalbench-import` (#97).
- Do not merge #456–#468.
- Do not start from `main`; stack on #467 head `a7fd389`.
- Do not restore `turn_index` — it is omitted on purpose in #467 until #429 exposes an importable full-conversation mapping. Do not fork #429's coordinate.

## Out of scope

- Merging #456–#468
- Preregistration seal / clock start / Week 1 job
- Dropping the EvalBench adapter (#97)
- SANA fork / new taxonomy names
- Live judge as denominator
- Public labeled corpus / Phase 3 golden set
- Benchmark harness (Phase 0)
- github.io / Pages demo updates
- Re-adding `turn_index` (#429)

## Related

- #435 (parent RFC)
- #466 (span-level G1 library issue)
- #463 (native `agent_events` writer)
- #429 (turn tagging / `sub_trajectories`; do not fork it)
- PRs #461, #464, #467, #468
- Demo (native path, EvalBench-format snapshot; not updated by this issue): https://caohy1988.github.io/evalbench/

Contributor guide

Open the contributing guide

Research direction

Start on #467 head a7fd389 by reading span_taxonomy.py and NativeAgentEventsRun.materialize, then compare the inherited EvalBenchRun.materialize path. Use the widget-stock fixture and offline unit tests to verify the three exact span-label rows, frozen eval_id identity, and fail-closed handling of missing span IDs without live BigQuery or production writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
data, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.