[Tracking] Identity-safe trace evaluation plan for #358-#360

---
title: Identity-Safe Trace Evaluation - Plan
type: fix
date: 2026-07-11
deepened: 2026-07-11
artifact_contract: ce-unified-plan/v1
artifact_readiness: implementation-ready
product_contract_source: ce-plan-bootstrap
execution: code
---

# Identity-Safe Trace Evaluation - Plan

## Goal Capsule

- **Objective:** Resolve issues #359, #358, and #360 in dependency order so trace retrieval, categorical judging, and the skill-evolution demo cannot silently mix reused session IDs that differ in identity or resolved scope, or bind a golden answer to the wrong conversation.
- **Authority:** The issue contracts and current repository behavior are authoritative. PR #351 remains example-scoped; its withdrawn SDK commits are reference implementations, not changes to restore wholesale.
- **Execution profile:** Land #359 first as the shared identity and scope contract, #358 second on that contract, and #360 only after its corresponding SDK capability is available. Prefer one reviewable change per issue.
- **Compatibility posture:** Preserve `session_id` and existing serialized fields. Add identity and provenance fields without breaking old callers, except that previously unsafe ambiguous singular lookups now return a typed error instead of silently choosing or merging data.
- **Stop conditions:** Stop and revisit the plan if producer data disproves the documented session/trace semantics, if the public response additions or the new fail-closed ambiguity behavior require a major-version contract, or if live BigQuery validation shows the selected scope cannot be reconstructed deterministically.
- **Tail ownership:** #360 owns demo and documentation cleanup. It must not be folded into either SDK-core change, and it must remain safe when #358 and #359 land at different times.

---

## Product Contract

### Summary

The SDK will distinguish a persistent conversation session from the scope used to select one recorded run of that session. Trace reads will carry a complete identity, apply caller-selected labels and experiment scope to the fetched rows, and reject ambiguous singular lookups. Categorical judge context will use the same identity across BigQuery-native generation, retry, and API fallback paths. Once each SDK capability lands, the skill-evolution lab can remove its corresponding workaround without overstating where judging or trace evidence came from.

### Problem Frame

Current trace selection and grouping use `session_id` as if it were globally unique. The repository's data model defines it only as a persistent conversation-thread identifier; users, root agents, experiments, and repeated evaluation passes can reuse it. `_LIST_TRACES_QUERY`, `_GET_SESSION_TRACE_QUERY`, trace construction, categorical transcript builders, and several downstream maps can therefore merge or overwrite distinct data.

Issue #358 adds trusted per-session judge context, but a context map keyed only by `session_id` would inherit the same collision. Issue #360 can remove the lab's per-slice tables and conversations-file judge only after both identity-safe trace reads and server-side judge context exist.

### Actors

- A1. **SDK developer:** retrieves traces and runs evaluations through the Python API.
- A2. **CLI or Remote Function caller:** requests one trace and needs an actionable ambiguity response when the session ID is reused.
- A3. **Agent or automation:** consumes serialized traces or categorical results and must not inspect or promote data from another run.
- A4. **Demo operator:** runs the skill-evolution lab and relies on exact-session gates, bounded BigQuery reads, and truthful artifact provenance.

### Requirements

**Identity and trace retrieval**

- R1. Define one public identity/selector contract shared by trace retrieval and categorical evaluation while retaining `session_id` as a compatibility field.
- R2. Treat `session_id` as the conversation identifier and keep producer `trace_id` as an execution-trace field; do not use either alone as the identity of a scoped multi-turn session.
- R3. Apply custom-label and experiment predicates to the outer event-row fetch, and apply `user_id`/`root_agent_name` identity with NULL-safe comparisons so selected sessions cannot absorb foreign rows.
- R4. Return enough identity and scope metadata for two traces with the same `session_id` to remain distinguishable after serialization.
- R5. Make singular lookups fail with a typed ambiguity result when more than one candidate remains; never choose the newest candidate as an implicit fallback.
- R6. Preserve complete-trace semantics: event-type, error, latency, and time filters select candidate sessions unless explicitly documented as row-scope filters.

**Judge context and evaluation parity**

- R7. Bind trusted judge context to the same identity used by trace selection, with legacy session-only keys accepted only for an unambiguous evaluated population.
- R8. Carry context unchanged through BigQuery `AI.GENERATE`, parse-error/NULL retry, and full Gemini API fallback without cross-session overwrites.
- R9. Skip `AI.CLASSIFY` when per-identity context is present and record the reason in report details so cost and latency changes are visible.
- R10. Aggregate exactly one transcript row per resolved identity before invoking `AI.GENERATE`, including batches with duplicate requested identities or no identities.
- R11. Add identity to categorical results and persistence deduplication without persisting raw golden answers or putting trace-derived values into BigQuery job labels.
- R12. Document per-identity context as trusted evaluator/reference material that is sent to BigQuery or Gemini through parameters, not interpolated into SQL, logged, persisted, or treated as untrusted conversation text.

**Agent-facing and demo integration**

- R13. Keep Python, CLI, Remote Function, GQL fallback, trace-evaluator, serialization, reporting scripts, and the self-monitoring agent example aligned with the identity and ambiguity contract.
- R14. Let #360 remove the #358 and #359 workarounds independently: shared-table trace enrichment must work while API judging remains, and server-side judging must work while the demo deliberately retains per-slice tables.
- R15. Keep the skill-evolution gate over the exact expected session set and require strict improvement; identity work must not weaken those protections.
- R16. Keep repository docs, committed sample artifacts, run banners, and the companion Gist truthful about which path supplied judging and which path supplied execution-span evidence.
- R17. Document the new ambiguity-error behavior in `CHANGELOG.md` for the #359 release, with migration guidance for callers that previously received data from ambiguous singular lookups (retry with explicit selectors or use the list APIs).

### Acceptance Examples

- AE1. **Cross-user collision:** Given two event populations with the same session ID but different users, an unscoped singular lookup returns an ambiguity error naming the dimensions required to retry; a user-scoped lookup returns only that user's rows.
- AE2. **Cross-root-agent collision with NULLs:** Given reused session IDs where one or both `root_agent_name`/`user_id` dimensions are NULL, NULL-safe matching keeps each candidate separate and never drops the all-NULL candidate.
- AE3. **Repeated evaluation passes:** Given V0 and V1 rows with the same session/user/root-agent identity but different run/slice labels, candidate resolution returns two scope signatures; an explicit run/slice selector returns only the chosen pass and its complete in-scope span tree.
- AE4. **Context isolation:** Given two resolved identities that share a session ID and have different expected answers, each judge path receives only its own context and produces a result carrying the corresponding identity.
- AE5. **Retry parity:** Given one `AI.GENERATE` parse failure, its retry receives the same context and identity as the initial call, while successful identities are not duplicated or re-evaluated.
- AE6. **Context without justification:** Given per-identity context and `include_justification=False`, evaluation starts at `AI.GENERATE` and reports that `AI.CLASSIFY` was skipped because context was required.
- AE7. **Partial #359 landing:** Given #359 is available and #358 is not, the lab uses one shared events table with run/slice-scoped trace enrichment while judging remains on the conversations/API path.
- AE8. **Independent #358 cleanup:** Given the server-side judging cleanup is applied while the table-layout cleanup is deliberately withheld, the lab uses server-side golden-grounded judging and still works with per-slice event tables.
- AE9. **Complete cleanup:** Given both SDK fixes, a shared-table run with repeated V0/V1 IDs produces the exact expected 80-session gate population, no mixed transcripts or span trees, and judge justifications grounded in the matched golden answer.

### Scope Boundaries

**In scope**

- Trace retrieval and evaluation surfaces that reconstruct or judge event rows from `agent_events`.
- Additive identity/provenance fields and the minimal persistence/view migration needed to avoid result collisions.
- Public and agent-facing selector parity across existing SDK surfaces.
- Conditional cleanup of the two PR #351 demo workarounds and their documentation.

**Deferred to follow-up work**

- A repo-wide conversion of every analytical `GROUP BY session_id`; session-level aggregation remains valid where the output intentionally represents a whole conversation rather than a scoped trace.
- A richer typed rubric/context framework beyond the trusted per-identity text required by #358.
- New approval, deployment, or scheduler workflows for evolved skills.
- A durable `docs/solutions/` note should be added after the behavior lands, when implementation evidence can document the final identity contract.

**Outside this plan**

- Reintroducing withdrawn SDK-core commits into PR #351.
- Changing producer schemas or requiring all producers to emit a new globally unique run identifier.
- Treating arbitrary BigQuery job labels as identity storage; repository policy forbids trace-derived job-label values.
- Distinguishing sessions reused with identical intrinsic identity and identical scope labels: such reuse remains a single indistinguishable candidate and its rows still merge. Producer-side run/slice labeling is the required disambiguation mechanism, since a globally unique run identifier is excluded above; documentation and release claims must state this residual limitation.

### Dependencies

- PR #351 supplies the base files and TODO markers for #360. A separate #360 follow-up branch may be prepared against PR #351's head before it merges, but no #360 cleanup belongs in PR #351 and the follow-up must not merge first.
- #359 must land before #358 so judge-context keys, transcript grouping, results, and retry maps share one identity contract.
- `google-cloud-bigquery>=3.0.0` already supports the required array-of-struct parameters and NULL-safe GoogleSQL operators; no dependency-floor increase is planned.
- Live BigQuery and model validation remain opt-in because they consume cloud resources.

---

## Planning Contract

### Key Technical Decisions

- KTD1. **Separate intrinsic identity from query scope.** Use immutable value objects with explicit semantics: `TraceIdentity(session_id, user_id, root_agent_name)`, `TraceScope(experiment_id, custom_labels)`, and `TraceSelector` for optional caller pins. A resolved selector combines the intrinsic identity with a canonical scope signature built from the full experiment/custom-tag payload, so a bare lookup can detect V0/V1 passes even when the caller did not already know their labels. Canonicalize custom-label keys into a sorted tuple before hashing/comparison. `trace_id` remains the producer's OpenTelemetry execution identifier because a multi-turn session may contain more than one trace ID. The existing public `TraceFilter` remains the list-level query surface and is refactored to construct and consume these value objects so exactly one alias-aware, NULL-safe predicate implementation exists; resolved selectors are outputs derived from fetched rows and inputs to singular/context APIs. U1 defines the mapping from `TraceFilter` fields to selector pins and states the long-term deprecation target, if any.
- KTD2. **Return identity, do not hide it.** `Trace` and `CategoricalSessionResult` gain an additive identity object while retaining legacy scalar fields. Serialization, CLI JSON, Remote Function output, and reporting retain the identity so downstream agents can retry or correlate safely.
- KTD3. **Fail closed for singular ambiguity.** A session-only singular request resolves candidate identities and scope signatures first. Multiple candidates produce a typed error whose printable form contains only the candidate count and retry dimension names; it must not dump user IDs, label values, event content, or judge context into logs. The structured/serialized error payload returned by the Remote Function, CLI JSON output, and the agent tool may carry candidate identity dimensions and scope signatures — never event content, judge context, or golden answers — so agents can retry in one step, while `__str__`/log rendering stays at count plus retry-dimension names. Authorized callers can use list APIs to retrieve fully identified results and retry explicitly. No timestamp-based winner is selected.
- KTD4. **Make row-scope predicates alias-aware.** Shared filter logic must be able to emit qualified predicates for the outer event alias. Labels and experiment scope are reapplied to event rows; `user_id` and `root_agent_name` participate in the NULL-safe anchor join. When supposedly config-level experiment/custom tags vary inside one resolved candidate, split candidates or fail closed rather than silently returning a mixed trace; a scoped trace is complete within the selected scope, not across excluded passes. Pass-splitting is the default; an explicit opt-in selector flag (e.g., `allow_mixed_scope`) returns the conversation-complete row set for one intrinsic identity with per-scope coverage metadata attached — the documented escape hatch for real conversations whose config tags drift mid-thread and that would otherwise be unreadable as a singular read.
- KTD5. **Keep identity batches deterministic.** Normalize and deduplicate requested identities before constructing BigQuery array-of-struct parameters. Use an explicit struct type for empty arrays so empty and non-empty batches have the same parameter contract and empty work never calls `AI.GENERATE`.
- KTD6. **Use identity-keyed judge context everywhere.** `per_session_context` accepts immutable resolved selectors (plus legacy session strings after uniqueness validation), and server-side generation joins parameterized context on that resolved identity. API/retry maps use the same key rather than `session_id`. A non-empty context mapping bypasses `AI.CLASSIFY` for the batch; unmapped identities still use `AI.GENERATE` with no appended context, avoiding a second split/merge path.
- KTD7. **Expose provenance without retaining answer keys.** Reports and persisted rows record resolved identity, execution mode, whether context was applied, and an SDK-defined context-source enum such as `golden_expected_answer`. Raw context, golden answers, caller-controlled source strings, and content-derived context fingerprints are not copied into result tables, serialized errors, logs, or job labels.
- KTD8. **Migrate persistence additively and in order.** First add nullable `user_id`, `root_agent_name`, `experiment_id`, canonical `scope_key`, versioned `identity_key`, context-applied, context-source, and execution-mode columns idempotently; then deploy writers that populate them; finally update views. Do not backfill historical rows because their lost scope cannot be reconstructed reliably. Views use the versioned identity key for new rows and an explicitly namespaced `legacy:` fallback for old rows, preserving legacy aggregates without merging them with newly scoped results. Straddle semantics: when a legacy row's `session_id` resolves to exactly one post-migration identity, the versioned row supersedes the legacy row in dedup views so re-evaluated sessions do not double-count; only legacy sessions with ambiguous resolution retain the never-merge fallback. Rollback reverts writers/views while leaving harmless nullable columns in place.
- KTD9. **Remove demo workarounds independently.** #359 cleanup restores a shared table plus scoped trace reads; #358 cleanup restores server-side judging plus expected-answer context. These are independently testable repository states, not permanent runtime feature flags. Either change must work while the other workaround remains.

### High-Level Technical Design

```mermaid
flowchart TB
A[Caller supplies session and optional scope] --> B[Resolve candidate identities]
B --> C{Exactly one for singular read?}
C -->|No| D[Return typed ambiguity with retry selectors]
C -->|Yes| E[Fetch rows with identity anchor and row scope]
B -->|List request| E
E --> F[Trace plus additive identity]
F --> G[Identity-keyed transcript and judge context]
G --> H{Evaluation path}
H -->|BigQuery| I[One AI.GENERATE row per identity]
H -->|Retry or fallback| J[Gemini API with the same identity and context]
I --> K[Identity-bearing categorical result]
J --> K
K --> L[Persist provenance without raw context]
L --> M[Remove matching demo workaround]
```

The identity resolver is the shared seam. Trace retrieval, categorical transcript construction, context binding, retries, and agent-facing serializers must consume its output rather than rebuilding session-only dictionaries independently.

### Sequencing and Landing Strategy

1. **#359 foundation:** introduce the contract, update trace SQL and construction, and propagate selectors/errors through all trace-retrieval surfaces.
2. **#358 evaluation context:** update categorical queries, result maps, fallbacks, persistence, and the quality-report caller using the #359 identity.
3. **#360 cleanup:** remove each workaround only when its predecessor is present, then run the shared-table server-side demonstration and refresh provenance.

Each phase should be reviewed and releasable on its own, with one caveat: U2 is independently reviewable but must not be released to end users ahead of U3, because CLI and Remote Function callers do not gain formatted ambiguity handling until U3 lands and would surface the new typed error as a raw exception. To shorten Eva's critical path, prepare them as a visible stack rather than waiting for each merge: #358 targets the reviewed #359 head, and #360 targets PR #351 plus the applicable reviewed SDK head. Rebase each follow-up onto the merged predecessor before landing. This keeps SDK-core changes out of PR #351 while allowing Eva to review interfaces, demo cleanup, and provenance wording early.

The first review checkpoint is the U1 public contract: exact model fields, hash/equality behavior, ambiguity payload (including the serialized-surface boundary — which identity dimensions structured responses carry versus the redacted printable form), and legacy-key rules. Once that checkpoint is accepted, U2-U3 and U4-U5 can advance without reopening the identity decision. Eva can update the companion Gist's current provenance wording immediately; the final server-side claim waits for the recorded U6 run.

### System-Wide Impact

- **Public API:** Singular trace retrieval gains ambiguity behavior; list/result objects and serialized responses gain additive identity fields.
- **Agent tools:** The CLI-backed self-monitoring agent and Remote Function must expose selectors and structured errors so agents can retry instead of guessing.
- **Evaluation correctness:** Every session-only dictionary seam can overwrite a collision even after SQL is fixed; transcripts, failed-session retries, resolved-response maps, golden metadata, and result persistence all require identity-keyed auditing.
- **BigQuery cost:** Duplicate identity structs can duplicate joins and model invocations. Deduplication and one-row-per-identity aggregation are correctness and cost requirements.
- **Privacy:** Identity and context are trace-derived data. They belong in normal result payloads under existing access controls, never in broadly visible job labels. Raw context must not be persisted by default.
- **Historical data:** Missing `user_id`, `root_agent_name`, label, or experiment fields must remain queryable through NULL-safe matching. Mixed historical tables may require explicit selectors and a clear partial-trace warning.
- **Documentation:** `docs/design.md` and `docs/hatteras_evaluation.md` currently teach session-only grouping and must be updated alongside the contract.

### Risks and Mitigations

- **Risk: treating `trace_id` as the session key splits multi-turn conversations.** Mitigation: preserve it as execution metadata and base scoped-session identity on documented session/user/root-agent semantics plus resolved scope.
- **Risk: full custom-tag payloads are not stable within a run.** Mitigation: treat producer config tags as scope-level metadata, canonicalize their key order, test missing/NULL tags, and fail closed or split candidates when the payload varies instead of choosing one row's tags.
- **Risk: grouping SQL is fixed but Python maps still overwrite collisions.** Mitigation: audit every `dict[session_id, ...]` seam and include two-identities/one-session tests for all evaluation modes.
- **Risk: label row scoping creates partial traces for historical rows with inconsistent tags.** Mitigation: document config-level tag expectations, split distinct scope payloads during candidate resolution, surface coverage warnings on list results, and fail a singular read when its selected scope is still internally inconsistent.
- **Risk: result-table schema drift breaks existing datasets or views.** Mitigation: use additive nullable columns in schema-writer-view order, never rewrite historical rows, compare pre/post row cardinality, and retain old views/writers as the rollback path.
- **Risk: context injection changes cost and latency by bypassing `AI.CLASSIFY`.** Mitigation: record the skip reason and execution mode in report details and cover the behavior with tests.
- **Risk: BigQuery SQL string tests pass while deployed SQL is invalid.** Mitigation: add an opt-in dry-run test with real query parameters plus a live row-matching test; dry runs prove syntax/type binding but not semantics.
- **Risk: #360 removes both workarounds at once.** Mitigation: encode and test the #358-only and #359-only states before the final combined state.
- **Risk: artifacts claim a data path they did not use.** Mitigation: derive README, run banner, scorecard, and Gist wording from recorded execution provenance and retain disclosure for historical hybrid runs.

### Sources and Research

- `src/bigquery_agent_analytics/client.py` contains the session-only list/get queries, fallback trace construction, categorical execution cascade, and session-only retry maps.
- `src/bigquery_agent_analytics/trace.py` defines `TraceFilter` and the public `Trace` model.
- `src/bigquery_agent_analytics/categorical_evaluator.py` contains the session-only transcript builders, categorical result models, and persistence DDL.
- `docs/design.md` defines `session_id` as a persistent conversation thread, `invocation_id` as one execution turn, and `trace_id` as the OpenTelemetry trace identifier.
- `docs/sdk_usage_tracking.md` prohibits trace-derived values in BigQuery job labels.
- Withdrawn commits `94ff0e7` and `da52c54` demonstrate label row scoping and server-side context plumbing, but do not cover the full identity contract or all fallback paths.
- [BigQuery Python `ArrayQueryParameter`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.query.ArrayQueryParameter) requires an explicit struct element type for an empty struct array.
- [GoogleSQL operators](https://cloud.google.com/bigquery/docs/reference/standard-sql/operators) define `IS NOT DISTINCT FROM` as NULL-safe and do not permit comparing whole structs with it.
- [BigQuery parameterized queries](https://cloud.google.com/bigquery/docs/parameterized-queries) support array and struct parameters; duplicate request structs still need caller-side normalization.
- [BigQuery JSON functions](https://cloud.google.com/bigquery/docs/reference/standard-sql/json_functions) do not preserve JSON-object key order; candidate tag payloads must be normalized into sorted scalar key/value tuples before they become scope keys.
- [BigQuery dry runs](https://cloud.google.com/bigquery/docs/running-queries) validate query parsing and parameter typing without proving row-level semantics.
- [BigQuery `AI.GENERATE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate) operates per input row, making one-transcript-row-per-identity an explicit cost and correctness invariant.

---

## Implementation Units

### U1. Define the identity, selector, and ambiguity contract

- **Goal:** Add the shared models and compatibility rules that every later unit consumes.
- **Requirements:** R1, R2, R4, R5, R13; AE1, AE2.
- **Dependencies:** None.
- **Files:** `src/bigquery_agent_analytics/trace.py`, `src/bigquery_agent_analytics/__init__.py`, `src/bigquery_agent_analytics/serialization.py`, `tests/test_sdk_trace.py`, `tests/test_serialization.py`.
- **Approach:** Introduce additive identity and selector value objects plus a typed ambiguity error. Keep existing `Trace.session_id`, `Trace.user_id`, and `Trace.trace_id` fields. Define which dimensions are intrinsic, which are caller-selected scope, how candidate selectors are safely exposed, how legacy session-only inputs are validated, and how `TraceFilter` fields map onto the new value objects behind a single shared predicate implementation.
- **Patterns to follow:** Existing dataclasses in `trace.py`, public exports in `__init__.py`, and recursive dataclass serialization in `serialization.py`.
- **Test scenarios:**
- Construct and serialize a trace with full identity and verify all legacy field names and values remain unchanged while the identity field is additive.
- Serialize identities containing NULL user/root-agent/scope fields and verify stable JSON output.
- Deduplicate equal selectors while keeping selectors that share a session ID but differ in identity or scope.
- Render a typed ambiguity error with safe retry-dimension hints and verify candidate values, raw context, and event content are absent from its printable form.
- Accept a legacy session-only key when exactly one candidate exists and reject it when multiple candidates exist.
- **Verification:** Public imports resolve, serialized output remains JSON-safe, and callers can distinguish collision candidates without parsing event rows.

### U2. Make trace SQL and construction identity-safe

- **Goal:** Resolve #359 in the core `Client` trace queries and row grouping.
- **Requirements:** R2, R3, R5, R6; AE1, AE2, AE3.
- **Dependencies:** U1.
- **Files:** `src/bigquery_agent_analytics/client.py`, `src/bigquery_agent_analytics/trace.py`, `tests/test_sdk_client.py`, `tests/test_sdk_trace.py`, `tests/test_pr16_fixes.py`, `tests/test_trace_identity_bigquery_live.py`.
- **Approach:** Select candidate identities with `user_id`, `root_agent_name`, and experiment/custom-tag metadata; normalize returned scalar custom tags in Python into a sorted key/value tuple rather than relying on JSON object order; then emit parameterized per-key predicates for the resolved row fetch instead of comparing whole JSON values. Anchor outer rows with NULL-safe composite conditions and alias-qualified row-scope predicates. Singular session requests resolve candidates before fetching and raise on ambiguity. Trace grouping uses the resolved selector rather than `session_id` alone. Normalize identity batches and give empty array-of-struct parameters an explicit element schema.
- **Execution note:** Start with collision characterization tests against the current session-only queries, then make the query builders and grouping satisfy them. Required precondition: query real `agent_events` data for per-session distinct-value counts of `user_id`, `root_agent_name`, `experiment_id`, and custom-tag payloads — including per-event-type NULL sparsity — and record the result as trigger evidence for the Goal Capsule stop condition before the anchor-join design is finalized; the design assumes these dimensions are row-uniform within one run, and synthetic fixtures cannot prove that.
- **Patterns to follow:** Parameterized query construction and labeled `QueryJobConfig` in `client.py`; opt-in live BigQuery tests such as `tests/test_materialize_window_live.py`.
- **Test scenarios:**
- Same session/different user returns two distinguishable list results and an ambiguity error for a bare singular read.
- Same session/different `root_agent_name` cannot leak rows across the NULL-safe anchor join.
- Same session/different run/slice labels returns only the explicitly selected pass.
- NULL user/root-agent values match only the intended NULL candidate.
- Same intrinsic identity with distinct full custom-tag payloads yields distinct scope candidates even before an explicit label selector is supplied.
- Session-selection filters still fetch the complete chosen trace, while row-scope filters exclude foreign-pass rows.
- Duplicate requested identities do not duplicate event rows; an empty identity batch returns no rows and runs no model/query work beyond validation.
- A real BigQuery dry run accepts empty and non-empty struct parameters; an opt-in live fixture proves actual row separation and deterministic ordering.
- **Verification:** List and singular reads satisfy collision cases, generated SQL dry-runs successfully, and no foreign identity or scope rows appear in returned traces.

### U3. Propagate selectors and ambiguity across public and agent-facing trace surfaces

- **Goal:** Prevent secondary APIs from bypassing #359 through session-only calls.
- **Requirements:** R4, R5, R13, R17; AE1, AE3.
- **Dependencies:** U2.
- **Files:** `src/bigquery_agent_analytics/cli.py`, `src/bigquery_agent_analytics/trace_evaluator.py`, `src/bigquery_agent_analytics/client.py`, `src/bigquery_agent_analytics/context_graph.py`, `deploy/remote_function/dispatch.py`, `scripts/quality_report.py`, `scripts/latency_report.py`, `examples/cli_agent_tool.py`, `docs/design.md`, `docs/hatteras_evaluation.md`, `CHANGELOG.md`, `tests/test_cli.py`, `tests/test_trace_evaluator.py`, `tests/test_context_graph.py`, `tests/test_remote_function.py`, `tests/test_quality_report_helpers.py`, `tests/test_latency_report_helpers.py`.
- **Approach:** Add selector inputs and structured ambiguity propagation to the CLI, Remote Function, GQL trace path and flat fallback, trace evaluator, reports, and agent example. Resolve a selector once, then make both GQL reconstruction and flat merging use that resolved selector; never run graph traversal on a broader session-only population. Make `BigQueryTraceEvaluator` reuse the shared resolver/query builder while preserving its async result model rather than maintaining a second identity algorithm. Replace downstream maps that collapse traces by session ID when they can receive more than one identity. Preserve simple session-only usage for unique datasets.
- **Patterns to follow:** Typer option/error handling in `cli.py`, serialized Remote Function results in `deploy/remote_function/dispatch.py`, and existing GQL fallback tests in `tests/test_context_graph.py`.
- **Test scenarios:**
- CLI and Remote Function session-only requests succeed for one candidate and return actionable ambiguity for multiple candidates.
- Explicit user/root-agent/run selectors reach the underlying client unchanged.
- GQL reconstruction and its flat SQL fallback use the same selector and cannot merge a different identity.
- `BigQueryTraceEvaluator` retrieves and evaluates only the resolved identity.
- Reporting scripts retain two traces sharing a session ID without overwrite.
- The ADK self-monitoring tool returns structured candidate selectors that an agent can use for a retry.
- **Verification:** Python, CLI, Remote Function, GQL, reporting, and agent-example outputs agree on identity and ambiguity semantics.

### U4. Add identity-bound judge context across every categorical path

- **Goal:** Resolve #358 without recreating session-only collisions in prompts, retries, or results.
- **Requirements:** R7, R8, R9, R10, R12; AE4, AE5, AE6.
- **Dependencies:** U1, U2.
- **Files:** `src/bigquery_agent_analytics/categorical_evaluator.py`, `src/bigquery_agent_analytics/client.py`, `tests/test_categorical_evaluator.py`, `tests/test_sdk_client.py`, `tests/test_client_labels.py`, `tests/test_ai_generate_judge_live.py`.
- **Approach:** Extend categorical transcript builders to aggregate by resolved identity. Join a parameterized context array on the same nullable identity fields, after deduplicating requests. Thread identity/context through server-side generation, failed-session retry, and full API fallback. Skip `AI.CLASSIFY` when context exists and record the skip reason. Keep prompt ordering equivalent across BigQuery and Gemini API paths.
- **Execution note:** Implement the API and fallback tests before changing the quality-report caller so the SDK contract is independently proven.
- **Patterns to follow:** Existing query builders and parse-error retry loop in `categorical_evaluator.py` and `client.py`; withdrawn `da52c54` only as a wiring reference.
- **Test scenarios:**
- Two identities sharing a session ID receive different context with no crossover in BigQuery SQL parameters or API prompts.
- Mixed batches apply context only to mapped identities.
- Parse-error and NULL retries keep the original identity/context and do not retry successful rows.
- Full API fallback keeps output order and identity while applying the same prompt context.
- Context plus `include_justification=False` bypasses `AI.CLASSIFY` and records why.
- Duplicate context identities result in one transcript/model call; empty context leaves the pre-existing query path unchanged.
- Non-empty `AI.GENERATE` status or NULL structured output follows the existing retry/error contract with identity retained.
- Captured log output across BigQuery `AI.GENERATE`, parse-error/NULL retry, and full API fallback execution contains no per-identity context text, golden-answer text, or context-source values.
- A live opt-in evaluation proves one model invocation and one result per identity.
- **Verification:** All three execution modes produce identity-equivalent results and prompt context, with execution-mode provenance visible and no extra model calls.

### U5. Preserve identity and provenance in reports, persistence, and views

- **Goal:** Prevent correct in-memory evaluation results from collapsing in quality reports or persisted categorical tables.
- **Requirements:** R4, R7, R11, R12, R13; AE4, AE5.
- **Dependencies:** U4.
- **Files:** `src/bigquery_agent_analytics/categorical_evaluator.py`, `src/bigquery_agent_analytics/categorical_views.py`, `src/bigquery_agent_analytics/client.py`, `scripts/quality_report.py`, `tests/test_categorical_evaluator.py`, `tests/test_categorical_views.py`, `tests/test_quality_report_helpers.py`.
- **Approach:** Add identity to categorical session results and replace session-only maps in quality reporting and retry reconciliation. Apply the additive schema migration before new writers, then update view deduplication to use the new versioned identity key with a namespaced historical fallback. Persist whether trusted context was applied, its SDK-defined source enum, and execution mode, never its raw text or a content-derived fingerprint. Keep historical rows unmodified and verify pre/post view cardinality before removing the rollback path.
- **Patterns to follow:** Idempotent BigQuery DDL in categorical persistence and current report-details provenance fields.
- **Test scenarios:**
- Reports retain two results sharing a session ID and attach each golden match to the correct identity.
- Retry reconciliation replaces only the failed identity, not every result with the same session ID.
- Existing categorical tables upgrade idempotently and historical rows remain visible through fallback deduplication.
- Re-evaluating a pre-migration session that resolves to exactly one post-migration identity yields a single row in "latest" views: the versioned row supersedes the legacy fallback row instead of double-counting.
- Deploying the schema, writer, and view in the required order never references a column before it exists; reverting writer/view code leaves old readers functional.
- New rows persist identity/provenance fields but omit raw expected-answer text.
- Views keep colliding identities separate and do not regress prompt-version deduplication.
- Generated report details distinguish AI.CLASSIFY, AI.GENERATE with context, and API fallback.
- **Verification:** In-memory, JSON, persisted-table, and dashboard-view representations preserve the same result cardinality and identity.

### U6. Remove demo workarounds and prove the end-to-end contract

- **Goal:** Resolve #360 after its prerequisites land and align all published provenance with the actual execution path.
- **Requirements:** R14, R15, R16; AE7, AE8, AE9.
- **Dependencies:** Merged PR #351; U3 for the #359 cleanup state; U5 for the #358 cleanup state; both for the combined state.
- **Files:** `examples/skill_evolution_lab/run_e2e_demo.sh`, `examples/skill_evolution_lab/compare_runs.py`, `examples/skill_evolution_lab/README.md`, `examples/skill_evolution_lab/VERIFICATION.md`, `examples/skill_evolution_lab/sample_run/README.md`, `examples/skill_evolution_lab/sample_run/run.log`, `scripts/quality_report.py`, `tests/test_quality_report_helpers.py`, `tests/test_compare_runs.py`, `tests/test_skill_evolution.py`. The sample log and compare-run test are supplied by the required PR #351 base, so they are intentionally absent from the current pre-merge checkout.
- **Approach:** Validate two independent cleanup states without introducing permanent capability flags. The #359 change collapses per-slice tables while retaining run/slice/time/limit filters for trace enrichment. The #358 change switches judging to the server-side path and passes identity-bound golden context while per-slice tables may remain. When both are present, run the combined shared-table/server-side workflow, retain exact-session and strict-win gates, and refresh artifacts only from recorded provenance. Coordinate the external Gist wording with Eva; do not claim an unrecorded path.
- **Execution note:** Exercise each partial-landing state before the combined live run; otherwise one cleanup can accidentally depend on the other.
- **Patterns to follow:** Existing TODO markers from PR #351, exact-session checks in `compare_runs.py`, and opt-in live-run documentation in the lab.
- **Test scenarios:**
- #359-only state uses one table, conversations/API judging, exact labels, and bounded trace enrichment.
- #358-cleanup-only state deliberately retains per-slice tables while using server-side golden-grounded judging.
- Combined state handles repeated V0/V1 IDs with no trace or context crossover.
- Missing expected sessions remain failures; stray sessions stay excluded; ties keep the incumbent.
- The recorded 80-session run has the expected slice counts and judge justifications reference the correct golden answer.
- README, VERIFICATION, sample README, run banner/log, and Gist all state the path recorded by the artifact.
- Historical hybrid samples retain a disclosure unless replaced by a new server-side recording.
- **Verification:** Both partial states and the combined state behave as specified; regenerated artifacts and external copy match recorded provenance; obsolete TODOs and workaround warnings are removed only when their replacement is live.

---

## Verification Contract

| Gate | Applies to | Evidence required |
|---|---|---|
| Focused identity tests | U1-U3 | `.venv/bin/python -m pytest tests/test_sdk_trace.py tests/test_sdk_client.py tests/test_serialization.py tests/test_cli.py tests/test_trace_evaluator.py tests/test_context_graph.py tests/test_remote_function.py tests/test_latency_report_helpers.py` passes. |
| Focused categorical tests | U4-U5 | `.venv/bin/python -m pytest tests/test_categorical_evaluator.py tests/test_sdk_client.py tests/test_client_labels.py tests/test_categorical_views.py tests/test_quality_report_helpers.py` passes. |
| Full repository suite | U1-U6 | `.venv/bin/python -m pytest` passes, with any pre-existing failure reproduced on the base commit and documented. |
| Formatting and whitespace | U1-U6 | `bash autoformat.sh` produces no unexpected diff and `git diff --check` is clean. |
| Shell validation | U6 | `bash -n examples/skill_evolution_lab/run_e2e_demo.sh` succeeds. |
| BigQuery dry run | U2, U4 | Identity and context queries parse with real empty/non-empty struct parameters and report bounded bytes processed. |
| Live collision fixture | U2-U5 | Opt-in BigQuery test proves reused IDs, NULL identities, run labels, retries, and result persistence do not cross-contaminate. |
| Live lab run | U6 | Shared-table/server-side run returns the exact expected session set, preserves the strict-win gate, and produces identity-consistent span trees and judge justifications. |
| Documentation audit | U3, U6 | Public API docs, design docs, README surfaces, committed sample provenance, and Gist wording match shipped behavior. |

Live gates are mandatory before closing their issues but remain opt-in for ordinary CI. Unit-level SQL/query-parameter construction stays in standard CI; real BigQuery dry runs and live/model calls require credentials and remain opt-in.

---

## Definition of Done

- #359 is complete when every existing trace-retrieval surface shares the identity/selector contract, row scoping is NULL-safe and alias-qualified, and ambiguous singular lookups fail with actionable retry-dimension hints instead of leaking candidate values.
- #358 is complete when trusted context binds to that identity across BigQuery generation, retry, and API fallback, and identity/provenance survive reporting and persistence without raw context retention.
- #360 is complete when each workaround can be removed independently, the combined demo passes its exact-session and strict-win gates on one shared table, and every artifact states its true data path.
- Existing callers using unique session IDs continue to work without changing arguments or parsing legacy fields.
- Unit, formatting, shell, dry-run, live collision, and live demo gates pass at their applicable phases.
- No session-only dictionary or persistence seam remains in the changed trace/evaluation paths where it can overwrite colliding identities.
- No trace-derived identity is added to BigQuery job labels, and no raw golden context is persisted by default.
- Documentation no longer teaches session-only trace identity, while intentional conversation-level aggregations remain clearly distinguished.
- Experimental helpers, withdrawn-commit remnants, obsolete TODOs, and abandoned migration paths are removed from the final diffs.

Start with issue #359, then review the trace retrieval queries, trace construction, categorical transcript builders, and downstream identity maps described here. Verify the selector and scope contract against agent_events and the existing Python, CLI, Remote Function, and evaluation entry points. Done means the dependent issues can land independently without mixed identities, ambiguous singular lookups, or cross-session judge context.

Written by the indexing model from the issue text.