GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories)

Open
#429 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

**The gap:** the flagship anti-parroting capability — parroted vs recovered corrections detected from execution traces — lives entirely in `scripts/quality_report.py`: the LLM turn tagger (`_tag_conversation_turns`), the `sub_trajectories` derivation (parroted/recovered/not_recovered with turn spans), and `_segment_trace_by_turns`. Three surfaces depend on its OUTPUT shape (judge reports, `skill_evolution`'s failure reclassification, the #395 host session schema), but no library surface produces it.

**Proposal:** a core `correction_analysis` module (or `trace_evaluator` extension) consuming identity-safe `Trace` objects (U1–U3) and producing identity-keyed correction segments — tagger, sub-trajectory derivation, trace segmentation — with the session-dict output shape (`sub_trajectories`, `execution_sub_trajectories` incl. per-segment `trace`, `correction_boundaries`, `verifications` int count) documented as the stable contract.

**Why core:** consumes core Traces, feeds core categorical results, and the parroting reclassification is the documented defense against fake successes. Precedent: #375 (U3) pulled `trace_evaluator` into the identity contract.

**Scope note:** larger than the sibling extractions (async tagging paths); the PR is staged after the golden-matching and rubrics extraction PRs establish the review pattern. Refs: #63, #395, #397.

Contributor guide

Open the contributing guide

Research direction

Start in scripts/quality_report.py, focusing on _tag_conversation_turns, sub_trajectories, and _segment_trace_by_turns. Read the identity-safe Trace requirements (U1–U3) and related issues #63, #395, and #397 before choosing the core module boundary. Done means a documented stable session-dict contract with identity-keyed correction segments, including the listed sub-trajectory fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.