[Design] U6 skill-evolution lab workaround removal and server-side recording

# Design: U6 skill-evolution lab workaround removal and server-side recording (#360)

**Status:** Approved (brainstorming and issue #384 review) — ready for implementation planning
**Plan unit:** #361 U6 (AE7, AE8, and AE9)
**Dependencies:** #358 and #359 closed; U3 and U5 merged on `upstream/main`
**Canonical companion Gist:** `evekhm/01c4673dddd68c1fd13383948eb7de35`

## Problem

The skill-evolution lab still carries two disclosed workarounds for SDK gaps
that are now closed:

1. It writes each scoring slice to a separate
`agent_events__` table because trace retrieval used to merge
reused session IDs across scopes (#359).
2. It judges `--conversations-file` input because the BigQuery/server-side
evaluator used to lack identity-bound per-session golden context (#358).

U6 removes both workarounds without replacing them with feature flags. It must
also prove the independently valid intermediate states required by #360/#361,
replace the historical hybrid `sample_run/` with a fresh fully server-side
recording, and update the canonical external Gist only after the repository
artifacts pass every gate.

## Goals

- AE7 proves one shared `agent_events` table while the API judge remains in
place temporarily.
- AE8 proves server-side golden-grounded judging on the already identity-safe
shared-table substrate.
- AE9 records the complete combined `--rounds 2` path with an exact
80-session held-out population, no V0/V1 mixing, and strict-win behavior.
- Each partial state has live evidence tied to its exact commit SHA.
- The committed sample, repository docs, run banner, provenance, PR
description, and canonical Gist tell the same story.
- Live verification uses expiring scratch data and never touches
`bqaa_hero_demo_20260708`.

## Non-goals

- No permanent runtime compatibility switches or feature flags.
- No reimplementation of identity selection, trace scoping, golden-context
matching, or persistence redaction in the example. Merged U3/U5 SDK surfaces
remain authoritative.
- No unrelated evaluator, dashboard, or skill-evolution algorithm changes.
- No substitute or forked Gist if the current credential cannot edit the
canonical `evekhm` Gist.
- No claim that the 500-session cap bound the normal 80-session demo. The
production command and report metadata attest the requested cap; a separate
cheap live probe proves cap behavior.

## Delivery shape and commit boundaries

The PR uses real commit boundaries rather than an atomic final-state patch.
Each boundary must be clean, independently testable, and cited by SHA.

### 0. Design and plan

The approved design and implementation plan land before runtime changes. They
do not count as AE7/AE8 evidence.

### 1. AE7 — shared table, post-U3 API path

- `run_agent()` writes every pass to one configured table, normally
`agent_events`.
- Rows retain collision-safe `run=` and `slice=` custom
labels.
- Judging temporarily remains on `--conversations-file`.
- BigQuery trace enrichment is bounded by app identity, exact run/slice
labels, `24h`, and `500`, and resolves through the merged U3 selector and
identity contracts. No session-only read path is reintroduced.
- Unit and shell contract checks pass at the AE7 commit.
- A complete default one-round live demo runs from a clean tree at the exact
AE7 commit SHA against an AE7 scratch dataset.

This is the #359-only cleanup state required by AE7.

### 2. AE8 — server-side judge on the AE7 substrate

- `score()` removes `--conversations-file` and evaluates the traces from
BigQuery.
- It keeps the app/run/slice/time/cap bounds introduced by AE7.
- Per-session golden context is matched and attached through the merged U5
identity-bound evaluator path.
- Unit and shell contract checks pass at the AE8 commit.
- A complete `--rounds 2` live demo runs from a clean tree at the exact AE8
commit SHA against a separate AE8 scratch dataset.

This is the #358-only cleanup applied on the already identity-safe substrate
and supplies the source artifacts for AE9. The AE8 exact-SHA gate run is the
one and only AE9 artifact-source run; U6 does not perform a third live demo on
a third scratch dataset.

### 3. AE9 — artifacts, docs, provenance, and combined gates

- The successful, exact-SHA AE8 `--rounds 2` gate run replaces `sample_run/`
wholesale.
- Generated report JSON/Markdown pairs, traffic files, skills, selections,
comparisons, and `run.log` all come from that one run.
- Repository docs and banners describe one shared table and fully server-side
golden-grounded judging.
- `sample_run/PROVENANCE.md` captures AE7, AE8, and AE9 evidence before scratch
teardown.
- The combined gates run against the complete artifact tree.
- Only a gated artifact commit may be pushed as the source for the Gist
update.

## Runtime architecture

### Shared write/read substrate

The demo resolves one event table once:

```text
..
```

Every session write carries:

- `root_agent_name=skill-evolution-lab`
- `user_id=lab-user`
- `custom_tags.run=`
- `custom_tags.slice=`

The same table is append-only for all passes in a run. Reused held-out session
IDs are expected and are separated by resolved identity plus trace scope.

### Bounded score selection

Each score request supplies:

- app/root-agent filter: `skill-evolution-lab`
- custom label: `run=`
- custom label: `slice=`
- time period: `24h`
- limit: `500`
- the existing eval spec and full dimensions
- turn tagging, trajectory enrichment, Markdown output, and JSON output

The final path supplies no conversations file. The traffic JSON remains a
diffable artifact and the expected-set source for comparison; it is not the
judge input.

### Exact held-out semantics

`compare_runs.py --questions` continues to define the expected held-out set:

- 55 single-turn questions
- 15 correction/anti-parroting questions
- 10 out-of-scope questions
- 80 total expected sessions per held-out pass

Missing sessions become failing placeholders. Stray sessions are excluded.
The candidate is retained only if its overall rate is strictly greater than
the incumbent; a tie keeps the incumbent.

## Scratch data lifecycle

- AE7 and AE8 use separate, uniquely named U6 scratch datasets. Names include
the phase and UTC run identifier. These are the only two live-demo datasets;
AE9 promotes the AE8 artifacts and does not create a third dataset.
- Every scratch dataset receives a seven-day default table expiration before
traffic is written. This keeps abandoned failure data available for
diagnosis while ensuring its tables self-delete.
- Scratch resources carry U6/phase labels when supported so they can be
identified without relying on name parsing.
- The standing `bqaa_hero_demo_20260708` dataset is explicitly out of scope and
must not be read, written, altered, or deleted.
- A failed run retains its active scratch data until the failure is diagnosed
and evidence is captured. A subsequent attempt uses a new dataset.
- Successful scratch datasets are deleted only after their durable evidence
has been written and checked. The provenance records teardown status.

## Durable provenance

`examples/skill_evolution_lab/sample_run/PROVENANCE.md` is the durable source
for live evidence. The PR description repeats its compact evidence table.

For AE7 and AE8 it records:

- phase and exact commit SHA
- UTC start/end timestamps
- project, scratch dataset, shared table, and run label
- command and demo mode (`--rounds 1` or `--rounds 2`)
- agent, analyst, and judge models and their distinct endpoints/regions
- report session counts and held-out exact-set results
- persisted event row counts grouped by run/slice
- distinct session/identity/scope counts
- repeated-ID isolation result
- timestamp and foreign-row exclusion result
- relevant query or verification output
- scratch teardown status

The synthetic bounds probe is phase-independent. It runs once per PR in the
populated AE8 scratch dataset and is recorded once as shared U6 evidence, not
as a required per-phase field for both AE7 and AE8.

For the final recording it additionally records:

- source AE8 SHA and artifact commit SHA
- hashes for key generated artifacts
- exact-80, missing-session, tie, and redaction gate results
- publication state

The current and fresh sample both use agent
`gemini-3.1-flash-lite` at Vertex `global`. Provenance states that the agent
model/endpoint is unchanged; the material migration is
hybrid-to-server-side judging and per-slice-to-shared storage. The judge is
recorded separately by model and region. If the live configuration actually
differs, the generated provenance and docs must report the observed values
instead of preserving an assumption.

### Gist publication states

Provenance supports exactly two terminal forms:

1. `gist_update: published`, with the canonical Gist revision URL/ID and the
SHA-256 of the published content.
2. `gist_update: prepared-but-blocked`, with the SHA-256 and local path of the
exact prepared replacement plus the permission error/owner-action needed.

The blocked state does not authorize a fork or substitute Gist.

## Live verification

### Persisted-data checks

Queries against the shared scratch table, not only in-memory report objects,
must establish:

- every expected slice exists under exactly one run label
- the live held-out slices contain the expected 80 distinct session IDs
- repeated V0/V1 IDs remain separate by scope and do not mix events,
transcripts, tool calls, or judge context
- no foreign run/slice/app rows enter the evaluated population
- selected rows fall inside the declared run and `24h` window
- event and identity cardinalities agree with the generated traffic and
report populations

The logged score command and report metadata prove `limit=500` was requested;
the persisted population proves it stayed below the cap.

### Cheap synthetic bounds proof

Once per PR, because the normal population never reaches 500, a dedicated,
non-demo label in the AE8 scratch table receives three recent traces, one
48-hour-old trace with the same label, and one recent trace with a foreign
label. A read-only selector requests the dedicated label, a `24h` window, and
a limit of `2`. It must return exactly two recent, correctly labeled resolved
traces and neither sentinel. This proves label, time, and limit enforcement
without creating or judging 500 synthetic sessions. The persisted fixture
counts and selector result are captured as shared U6 provenance.

### Server-side golden grounding and safety

The AE8/AE9 report checks establish:

- each in-scope session resolves to the correct golden Q&A through its exact
identity/scope selector
- judge justifications use the matched expected answer on the server-side path
- contextual payload fields do not escape through persisted evaluator rows or
failure messages, relying on the merged U5 redaction contract
- arbitrary context-source values remain rejected

The U5 focused tests remain part of the verification set; U6 does not copy
their logic.

## Repository changes

### `examples/skill_evolution_lab/run_e2e_demo.sh`

- Remove both `TODO(#360)` workarounds.
- Use the shared table for writes and reads.
- Apply app/run/slice/time/cap bounds.
- Remove `--conversations-file` from final scoring.
- Update banner/provenance text to the final path.
- Keep generated traffic files as artifacts and comparison inputs.

### Documentation

Update:

- `CHANGELOG.md`
- `examples/skill_evolution_lab/README.md`
- `examples/skill_evolution_lab/VERIFICATION.md`
- `examples/skill_evolution_lab/sample_run/README.md`
- generated `sample_run/run.log` and scorecards

The CHANGELOG closes the U4/U5 release-notes arc with one entry stating that
the #358/#359 demo workarounds are removed and the example now exercises the
shared-table, server-side path end to end.

Remove obsolete #358/#359 limitation language everywhere. Historical hybrid
disclosures may disappear only because the historical sample is being replaced
by a real fully server-side recording.

### `scripts/quality_report.py`

U3/U5 already own identity-safe selection and per-session context. U6 adds no
parallel implementation. The AE7 API-path trajectory enrichment must preserve
the CLI app/run/slice/time bounds when it delegates to the merged U3
`TraceFilter`; it may replace only the requested session IDs and display
limit, and must not mutate the caller's base filter. This is filter plumbing,
not a second identity-selection path. Any surviving workaround warning or
KNOWN LIMITATION text is removed; otherwise the scoring runtime remains
unchanged.

### Tests

Extend the focused suites rather than adding a feature-flag matrix:

- `tests/test_quality_report_helpers.py`
- `tests/test_compare_runs.py`
- `tests/test_skill_evolution.py`

Tests cover the final shell contract, absence of workaround text, expected-set
and stray semantics, missing-session failures, strict tie rejection, and the
existing identity/golden-context contracts.

## Error and publication behavior

- BigQuery logging failure is fatal.
- A server-side evaluation, golden match, cardinality, persisted-data, or
report-generation failure is fatal.
- An incomplete canonical recording (including fewer than 80 held-out
sessions) is not promoted to `sample_run/`.
- A failed gate cannot update the Gist.
- Scratch data remains for diagnosis until evidence capture, then is torn down
explicitly; seven-day table expiration covers abandoned runs.
- Gist write failure produces the prepared-but-blocked provenance state and an
exact owner handoff. It does not silently succeed via another account or
location.

## Publication sequence

1. Produce the exact-SHA AE8 `--rounds 2` gate run in scratch. This same run,
rather than a later third run, is the AE9 artifact source.
2. Cross-check persisted data and generated reports.
3. Replace `sample_run/`, update docs, and write provenance.
4. Commit the AE9 artifact tree.
5. Run combined offline and live gates.
6. Push the gated artifact commit and open/update the PR so its artifacts have
stable links.
7. Prepare the Gist replacement from that exact commit and hash it.
8. Update the canonical Gist and verify the returned revision, or record the
prepared-but-blocked state.
9. Commit/push the final provenance status and rerun affected documentation
and focused tests.
10. Delete successful scratch datasets after evidence capture is verified.

## Acceptance checks

### AE7 exact-SHA gate

- Clean AE7 commit.
- Focused unit/shell checks green.
- Full default one-round live demo green in expiring scratch.
- Shared table confirmed from persisted data.
- API judging and identity-safe U3 trace enrichment confirmed.
- SHA, run ID, dataset/table, commands, and results retained for AE9
provenance.

### AE8 exact-SHA gate

- Clean AE8 commit.
- Focused unit/shell checks green.
- Full `--rounds 2` live demo green in separate expiring scratch.
- Server-side judge consumes identity-bound golden context.
- Shared-table repeated IDs do not mix.
- Exact 80 held-out sessions per pass.
- SHA, run ID, dataset/table, commands, queries, and results retained for AE9
provenance.

### AE9 combined gate

- `bash -n examples/skill_evolution_lab/run_e2e_demo.sh`
- focused quality-report, compare-runs, and skill-evolution tests
- merged U3/U5 identity, provenance, redaction, and live-collision tests
- full offline suite
- formatter/lint/diff checks required by the repository
- persisted-data cardinality and isolation checks
- synthetic label/`24h`/`limit=2` proof
- missing-session probe fails safely
- stray-session probe excludes the stray
- tie probe keeps the incumbent
- all generated JSON/Markdown/report/log/provenance claims agree
- canonical Gist update occurs only after the above checks pass

## Definition of done

U6 is complete when AE7 and AE8 each have attributable exact-SHA live evidence,
the final demo uses one shared `agent_events` table and the server-side
golden-grounded judge, a fresh complete `--rounds 2` sample and docs are
truthful, all gates pass against scratch data, the hero dataset is untouched,
scratch teardown is recorded, and the canonical Gist is either updated or has
an exact hashed owner-ready update recorded as the sole external blocker.

Start with examples/skill_evolution_lab/run_e2e_demo.sh and the existing compare_runs.py --questions flow, then review the listed README, VERIFICATION.md, CHANGELOG.md, and sample_run artifacts. Use the unit, shell-contract, U5-focused, live verification, and combined gates to validate each commit boundary. Done means AE7 and AE8 have exact-SHA evidence, AE9 promotes the AE8 run into a consistent sample_run with provenance, and canonical Gist publication reaches one of the stated terminal states.

Written by the indexing model from the issue text.