GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

Span-level G1 taxonomy on span_id (AgentForensics RFC gap)

Open
#466 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

Parent: #435 (RFC: AgentForensics collaboration — automated failure taxonomy and **span-level** trace auditing).

This is the next engineering slice after the native `agent_events` writer #463 (PRs #464 native writer + #465 native freeze e2e). #455 is merged; #456–#465 are open freeze/native evidence with agent review complete (or stacked on it) and **must not be merged as part of this work** — do not wait for org APPROVE of #456–#465 to start.

## The gap

#435 says it plainly under "What is genuinely missing": **classification is session-level, not span-level.** The RFC's Phase 2 asks to extend classification from session to span/turn granularity, emitting `(trace_id, span_id, failure_category, evidence, confidence)`, reusing the #429 turn tagging / `sub_trajectories` work rather than inventing a parallel span model.

AgentForensics' core claim is locating *the exact failure point*. Today the frozen pipeline can say "session `7e352c34` failed with `task/planning`, `finalization`, `tool blockers`" — but not *which span* died. That attribution to a `span_id` is the missing localization layer, and the native writer already carries the fields for it: `native_events.py` passes `trace_id`, `span_id`, and `parent_span_id` through from production ADK `agent_events` rows.

## Goal

Attribute frozen G1 labels to the *span* that failed on the widget-stock silence session `7e352c34` (full id `7e352c34-4c1c-4395-acd5-fb3c8f215346`), not only the session row.

- **Source of truth is production ADK `agent_events` (the native path).** The EvalBench adapter (#97) stays optional; this slice must not require EvalBench source `configs` / `results` / `scores`.
- **Session-level `failed_sessions` + G1 from #461/#464 remains the denominator.** Span-level labels are a localization layer on top, not a replacement denominator.
- **Frozen G1 names stay exactly as frozen** (`failure_taxonomy.py`, `taxonomy_version: 0.1.0`, `g1_frozen: true`): the mechanical mapper emits `task/planning`, `finalization`, `tool blockers`. No SANA fork and no new category strings in this slice. If a test proves a span cannot be labeled with the frozen three, preserve the failing case and defer any taxonomy change to a separate issue.
- **Reuse #429 turn tagging and `sub_trajectories`** for span/turn structure. Do not invent a parallel mechanism.

## Acceptance (fixture-testable, no live BQ)

On the widget-stock fixture session `7e352c34`:

1. Session-level G1 still yields `task/planning`, `finalization`, `tool blockers` — unchanged.
2. At least one span is labeled. The known punchline — the trace dies after `AGENT_STARTING`, before `check_inventory` is ever called ("Real ADK events: USER_MESSAGE_RECEIVED -> INVOCATION_STARTING -> AGENT_STARTING, then silence") — becomes inspectable as a localized row: `span_id` + frozen category + evidence. For the silence case, the intended target is the last existing span (`AGENT_STARTING`), with evidence stating that no subsequent `TOOL_STARTING` / `check_inventory` event occurred — or the PR may explicitly choose between that attachment and a synthetic gap marker. Either way, no synthetic span identifiers: a gap marker, if used, must remain evidence anchored to a real native `span_id` from `agent_events`, not invent a new span id.
3. Output shape includes `span_id` (and `trace_id` when present on the events, as it is on the native rows). The frozen identity rule stays as-is: `eval_id = session_id[:8]` when that first-eight value is unique among sessions in the snapshot / join set; if the first-eight value collides, the full `session_id` is used as `eval_id`. No new identity scheme beyond this frozen first-8-with-full-session-id-on-collision rule — span labels must stay joinable to the same `eval_id` the `failed_sessions` / G1 contract already uses.
4. Offline unit tests only: no live BigQuery, no `--synth`, no live/LLM judge, and never any writes to production `agent_events`.

## Guardrails

- **Clock stays OFF.** This issue does not start the six-week clock, seal the preregistration, or kick a Week 1 snapshot job.
- **D4 fail-closed:** consumer remains Hai-Yuan Cao only.
- Never write production `agent_events` (the native writer's destination validation rejecting the reserved name stays in force).
- Do not delete or rewrite the `evalbench-import` adapter (#97).
- Do not merge #456–#465.
- **Stacking:** the frozen G1 / native writer files live on the #464 stack, not on GitHub `main`. The future PR for this slice stacks on #464 head `f5c6109` (not on #465 demo commits, not on #462). GitHub showing #458–#464 files vs `main` is expected stacking, not a finding.

## Out of scope

- Merging #456–#465
- Preregistration seal / clock start / Week 1 snapshot job
- Dropping the EvalBench adapter (#97)
- SANA fork or new taxonomy names
- Live/LLM judge as denominator
- Public labeled corpus / Phase 3 golden set
- Benchmark harness (Phase 0)

## Related

- #435 — parent RFC (Phase 2: span/turn granularity)
- #429 — turn tagging / `sub_trajectories`
- #463 — native `agent_events` writer (this slice's predecessor)
- PRs #461 (Week 0 G1 freeze, v0.1.0), #464 (native writer, head `f5c6109`), #465 (native freeze e2e, head `2169889`)
- Demo (EvalBench-format snapshot of frozen Week 0 / native path): https://caohy1988.github.io/evalbench/

Contributor guide

Open the contributing guide

Research direction

Start with failure_taxonomy.py and native_events.py on the #464 head, then trace the #429 turn-tagging and sub_trajectories work. Run the offline fixture-based unit tests for session 7e352c34. Done means frozen session labels remain unchanged and at least one native span has a span_id, category, and evidence tied to the existing eval_id without live BigQuery or production writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
data, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.