Azure / Azure/agentops

feat(doctor): change attribution, explain what changed since the last comparable run

Open
#494 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13
Forks
11
Avg merge
3h 20m
Merged PRs (30d)
42

Description

## Summary

Doctor can tell you that a metric regressed, but it gives you no way to see **why**. Today the only path is to manually dig through git history and guess which prompt or agent change lined up with the drop.

This issue proposes a **change-attribution** feature: record the git commit behind every eval run, then use that to diff the prompt file and agent target between the latest run and its most recent comparable baseline. The result surfaces in two places:

1. A new deterministic Doctor check (`change_attribution.prompt_diff`) that reports the change in plain language.
2. A new Cockpit page (`/api/version-timeline`) showing the same story across the whole run history, newest first.

No LLM in the loop — the whole thing is deterministic, cheap, and unit-testable without a m

## Motivation

- A regression finding wiser to leave the tool toinvestigate.
- We already store enough this possible; we justdon't record *which commit* produced the run.
- Both Doctor and Cockpitnged" means, so they needto share one diff implementation.

## Proposed design

### 1. Record the git SHA on every eval run

New helper `agentops/core/git_info.py::resolve_git_sha(workspace=None)`:

- Checks CI env vars first (`GITHUB_SHA`, `BUILD_SOURCEVERSION`,
`Build.SourceVersion`) siys correct in a pipeline.
- Falls back to `git rev-parse HEAD` for local development.
- Never raises: missing `or timeout all return `""`.

Every run writer adds two→ `config`:

| Field | Value |
|---|---|
| `git_sha` | commit behinknown |
| `prompt_file` | the prompt file path from the agent config, or `null` |

Touched writers: `pipeline/orchestrator.py` (local + cloud snapshots),
`pipeline/azd_runner.py`,.py`.`pipeline/prompt_deploy.py::_git_sha()` is refactored to reuse the same helper
(it previously only read

### 2. Expose the new fie

`RunSummary` gains `git_s`, and a new`dataset_evaluators_fingerprint`.

The new fingerprint hashes **(dataset, evaluators)** only — deliberately
excluding the agent targethodology_fingerprint`. The regression check needs the target to match; the attribution check needs to
compare runs *across* an e explaining that bump isthe whole point. It still refuses to compare runs that used a different
dataset or evaluator set, be misleading.

`_methodology_fingerprint_dataset_and_evaluators()`and `_hash_fingerprint_payload()` with the new one — no behaviour change.

### 3. New Doctor check: `change_attribution.prompt_diff`

`agentops/agent/checks/change_attribution.py`:

- Takes the latest run and its most recent baseline with the same `dataset_evaluators_finge
- Bails out silently (no finding) when: fewer than `min_runs`, no compbaseline, either run has ourced runs), or both runsshare the same SHA. - Runs `git diff ` (best-effort, 10 stimeout, returns `None` on any failure so a Doctor run can never crashgit is unavailable or his
- Emits one `INFO` finding under `Category.QUALITY` with a plain-langusummary, e.g.:
> Between these runs, t(s) added, 4 removed) andthe agent target changed: my-agent:1.2.0 -> my-agent:1.3.0.
Evidence carries `latest_run_id`, `baseline_run_id`, `old_git_sha`, `new_git_sha`, `diff_summs to the exact `git diff`command to run.
`summarize_prompt_diff()` is a pure function so it's testable without a real git repo.
Config (`agentops.agent.cattribution`):
```yaml
checks: change_attribution:
enabled: true min_runs: 2
prompt_paths: [] # files to diff, relative to workspace; falls back to the run's own prompt_file

Registered in checks/cataults_history", "workspace") and wired into analyzer.analyze() right after the regression check.

4. New Cockpit page: version timeline

GET /api/version-timeline renders the last 50 runs, newest first. Each row
shows pass/fail badge, ru), timestamp, target, short SHA, and a one-line change summary vs. the run immediately before it:

- "First recorded run."
- "Same commit as the pre
- "No commit recorded for one of these runs - re-run agentops eval run to
start tracking changes.
- Otherwise the same summarize_prompt_diff() sentence as the Doctor check.

It reuses _git_diff_stat / summarize_prompt_diff from the check so the two
surfaces never disagree. mplate gets a heading slotand a "version timeline" link in the header; _project_run() now also projects
git_sha, prompt_file, and

Scope of the spike

Files touched (≈730 inser

- src/agentops/core/git_i
- src/agentops/agent/checks/change_attribution.py (new) src/agentops/agent/checzer.py, agent/config.py
- src/agentops/agent/sources/results_history.py src/agentops/agent/cock
- src/agentops/pipeline/{orchestrator,azd_runner,azd_eval_runner,prompt_d}.py
- tests/unit/test_agent_checks_change_attribution.py (new), tests/unit/test_cockpit
t of scope / follow-ups
LLM-generated explanatif means semantically (thischeck is intentionally deterministic; an llm_assist variant could layertop later).
- Diffing anything beyond the prompt file and agent target (datasets, evaluator configs, thre
- Attribution for cloud-sourced runs that carry no git_sha. Rendering the actual dintly only the line-countsummary).
Plan
- [ ] Split the spike into reviewable commits (git SHA recording → resulthistory → Doctor check
- [ ] Docs: new check in the Doctor checks reference, change_attribution config block, Cockpit t
- [ ] Decide whether prompt_paths should also accept globs
- [ ] Open PR against dev

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the proposed flow in pipeline/orchestrator.py, pipeline/azd_runner.py, results_history.py, and agent/checks/change_attribution.py. Read the new tests in tests/unit/test_agent_checks_change_attribution.py and the Cockpit tests first. Done means git SHAs and prompt files are recorded, the deterministic Doctor check and /api/version-timeline share the diff summary, and the documented tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
api, backend, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.