GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK
Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories)
- Dominant language
- Python
- Stars
- 47
- Forks
- 21
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 33
Description
**The gap:** the flagship anti-parroting capability — parroted vs recovered corrections detected from execution traces — lives entirely in `scripts/quality_report.py`: the LLM turn tagger (`_tag_conversation_turns`), the `sub_trajectories` derivation (parroted/recovered/not_recovered with turn spans), and `_segment_trace_by_turns`. Three surfaces depend on its OUTPUT shape (judge reports, `skill_evolution`'s failure reclassification, the #395 host session schema), but no library surface produces it.
**Proposal:** a core `correction_analysis` module (or `trace_evaluator` extension) consuming identity-safe `Trace` objects (U1–U3) and producing identity-keyed correction segments — tagger, sub-trajectory derivation, trace segmentation — with the session-dict output shape (`sub_trajectories`, `execution_sub_trajectories` incl. per-segment `trace`, `correction_boundaries`, `verifications` int count) documented as the stable contract.
**Why core:** consumes core Traces, feeds core categorical results, and the parroting reclassification is the documented defense against fake successes. Precedent: #375 (U3) pulled `trace_evaluator` into the identity contract.
**Scope note:** larger than the sibling extractions (async tagging paths); the PR is staged after the golden-matching and rubrics extraction PRs establish the review pattern. Refs: #63, #395, #397.
Contributor guide
Research direction
Start in scripts/quality_report.py, focusing on _tag_conversation_turns, sub_trajectories, and _segment_trace_by_turns. Read the identity-safe Trace requirements (U1–U3) and related issues #63, #395, and #397 before choosing the core module boundary. Done means a documented stable session-dict contract with identity-keyed correction segments, including the listed sub-trajectory fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100