GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

examples: simplify agent_improvement_cycle and self_evolving_agent_demo now that rubrics, golden matching, and evolution hooks are in core (#431/#432/#395)

Open
#482 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

Now that the canonical evaluation rubrics (#431), golden Q&A matching (#432), and the skill-evolution host hooks (#395) live in core, two older examples still carry hand-rolled copies of that functionality. `examples/skill_evolution_lab/` already imports the engine and scores via `quality_report.py` — no changes needed there.

## examples/agent_improvement_cycle (~130 removable lines)

- `agent_improvement/prompts.py` `JUDGE_PROMPT` and the `{tool_check}`/`{tool_fail_rule}` slots in `eval_runner.py` are lossy paraphrases of the canonical `response_usefulness` and `tool_usage` rubrics in `src/bigquery_agent_analytics/evaluation_rubrics.py`. Replace the bespoke `judge_case()` judge call with `build_metrics()` + `Client.evaluate_categorical`.
- `improver_agent.py` `build_report_from_eval_results()` exists only to fake canonical categories back out of a binary pass flag — delete once the judge emits canonical categories.
- `improver_agent.py` hand-partitions sessions and hand-formats trajectories; replace with `skill_evolution.partition_trajectories()` / `format_trajectory()` (the latter now renders tool calls and correction evidence).

**Open decision:** the canonical `response_usefulness` has three categories; the LoopAgent exit condition keys off a binary `pass`. A `partial` → pass/fail mapping policy must be chosen (mapping `partial` to fail makes some currently-passing cycles iterate longer). Also: canonical judging expects traces in BigQuery, but `run_eval.py --golden` is explicitly a no-BQ path — either keep a local-conversation judging path or accept a BQ dependency on that flag.

## examples/self_evolving_agent_demo (~100 lines net)

- `analyze_and_evolve.py` is a second, weaker evolution engine: `_generate_candidate_prompt()` (one-shot rewriter), the hand-rolled promotion gate, and `_write_prompt_diff()` are all superseded by `evolve_skill()` / `select_candidate(..., incumbent_score=)` / `evolve_skill(artifacts_dir=...)`.
- `analytics/session_metrics.py` `load_quality_summary()` aggregates a deterministic `quality_passed` stand-in for `response_usefulness`+`tool_usage`, superseded by the canonical rubric. (`run_sdk_evaluators()` is already correct SDK usage; keep.)

**Open decision:** this demo's narrative is *operational-waste-driven* evolution (token waste, tool-call overuse), not quality-failure analysis. Routing it through `evolve_skill()` risks flattening that distinct story — confirm intent before refactoring. The report-shape adapter it needs (`quality_passed` → `sessions[].metrics.response_usefulness.category`) is the same bridge `agent_improvement_cycle` has; write it once, share it.

## examples/README.md

Three demos now tell overlapping "agent improves its own prompt" stories with no stated differentiator. Add one line each naming what is distinct: LoopAgent tool-driven loop / operational-waste signal / golden-graded skill evolution.

Contributor guide

Open the contributing guide

Research direction

Read the listed files under examples/agent_improvement_cycle, examples/self_evolving_agent_demo, and examples/README.md, then compare their local logic with evaluation_rubrics.py and the skill_evolution APIs. Run the affected demos to understand the golden/no-BigQuery and operational-waste paths. Done means duplicated evaluation and evolution code is removed, the open behavior decisions are resolved, and the README distinguishes the three demos.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, developer-experience
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.