Azure / Azure/azure-sdk-tools

[azsdk-cli code review] Build the before-and-after evaluation harness

Open
#16,767 0 comments 0 reactions 1 assignee Claimed by @samvaity View on GitHub
Central-EngSys enhancement
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

## Parent

- #16764

## Goal

Build a frozen, reproducible before/after benchmark that measures whether the azsdk-cli code-review skill improves review quality rather than merely changing review output.

## Deliverables

- [ ] Select at least 8 maintainer-approved replayable diffs: 4 defect-bearing and 4 clean/analyzer-only controls.
- [ ] Cover behavioral, cross-file integration, reliability/concurrency, and error/response-contract defects.
- [ ] Record expected category, changed line, failure path, and analyzer/CI ownership for each fixture.
- [ ] Capture the no-skill baseline using a fixed model and review effort.
- [ ] Run the skill treatment against identical fixtures and settings.
- [ ] Run 3 repetitions per fixture for the final comparison.
- [ ] Preserve `results.jsonl`, comments, skill/tool trajectories, grader output, and raw counts as artifacts.
- [ ] Add deterministic grading for category + location, using a custom grader only if existing Vally graders are insufficient.
- [ ] Publish a before/after scorecard and rejection reasons.

## Metrics

| Metric | Target |
|---|---|
| Defect recall | >= 80%, no baseline regression |
| Review precision | >= 80% |
| Evidence rate | 100% |
| Duplicate analyzer/CI finding rate | <= 10% |
| Unrelated-change scope accuracy | 100% |
| Repeated-run stability | >= 75% |

Use raw numerator/denominator counts alongside percentages. Classify rejected findings as `false-positive`, `duplicate`, `unsupported`, `stale-rule`, `wrong-line`, or `unclear-impact`.

## Acceptance criteria

- Another maintainer can reproduce baseline and treatment results from documented commands.
- Fixtures and labels are frozen before final treatment scoring.
- Artifacts explain both what improved and what did not.
- Confirmed misses and false positives can be converted directly into regression fixtures.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.