GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK
examples: simplify agent_improvement_cycle and self_evolving_agent_demo now that rubrics, golden matching, and evolution hooks are in core (#431/#432/#395)
- Dominant language
- Python
- Stars
- 47
- Forks
- 21
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 33
Description
Now that the canonical evaluation rubrics (#431), golden Q&A matching (#432), and the skill-evolution host hooks (#395) live in core, two older examples still carry hand-rolled copies of that functionality. `examples/skill_evolution_lab/` already imports the engine and scores via `quality_report.py` — no changes needed there.
## examples/agent_improvement_cycle (~130 removable lines)
- `agent_improvement/prompts.py` `JUDGE_PROMPT` and the `{tool_check}`/`{tool_fail_rule}` slots in `eval_runner.py` are lossy paraphrases of the canonical `response_usefulness` and `tool_usage` rubrics in `src/bigquery_agent_analytics/evaluation_rubrics.py`. Replace the bespoke `judge_case()` judge call with `build_metrics()` + `Client.evaluate_categorical`.
- `improver_agent.py` `build_report_from_eval_results()` exists only to fake canonical categories back out of a binary pass flag — delete once the judge emits canonical categories.
- `improver_agent.py` hand-partitions sessions and hand-formats trajectories; replace with `skill_evolution.partition_trajectories()` / `format_trajectory()` (the latter now renders tool calls and correction evidence).
**Open decision:** the canonical `response_usefulness` has three categories; the LoopAgent exit condition keys off a binary `pass`. A `partial` → pass/fail mapping policy must be chosen (mapping `partial` to fail makes some currently-passing cycles iterate longer). Also: canonical judging expects traces in BigQuery, but `run_eval.py --golden` is explicitly a no-BQ path — either keep a local-conversation judging path or accept a BQ dependency on that flag.
## examples/self_evolving_agent_demo (~100 lines net)
- `analyze_and_evolve.py` is a second, weaker evolution engine: `_generate_candidate_prompt()` (one-shot rewriter), the hand-rolled promotion gate, and `_write_prompt_diff()` are all superseded by `evolve_skill()` / `select_candidate(..., incumbent_score=)` / `evolve_skill(artifacts_dir=...)`.
- `analytics/session_metrics.py` `load_quality_summary()` aggregates a deterministic `quality_passed` stand-in for `response_usefulness`+`tool_usage`, superseded by the canonical rubric. (`run_sdk_evaluators()` is already correct SDK usage; keep.)
**Open decision:** this demo's narrative is *operational-waste-driven* evolution (token waste, tool-call overuse), not quality-failure analysis. Routing it through `evolve_skill()` risks flattening that distinct story — confirm intent before refactoring. The report-shape adapter it needs (`quality_passed` → `sessions[].metrics.response_usefulness.category`) is the same bridge `agent_improvement_cycle` has; write it once, share it.
## examples/README.md
Three demos now tell overlapping "agent improves its own prompt" stories with no stated differentiator. Add one line each naming what is distinct: LoopAgent tool-driven loop / operational-waste signal / golden-graded skill evolution.
Contributor guide
Research direction
Read the listed files under examples/agent_improvement_cycle, examples/self_evolving_agent_demo, and examples/README.md, then compare their local logic with evaluation_rubrics.py and the skill_evolution APIs. Run the affected demos to understand the golden/no-BigQuery and operational-waste paths. Done means duplicated evaluation and evolution code is removed, the open behavior decisions are resolved, and the README distinguishes the three demos.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, developer-experience
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100