Azure / Azure/azure-sdk-tools

[azsdk-cli code review] Run a shadow-review pilot and decide auto-review

Open
#16,768 3 comments 0 reactions 1 assignee Claimed by @samvaity View on GitHub
Central-EngSys enhancement
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

## Parent

- #16764

## Goal

Validate the skill on real azsdk-cli pull requests in non-blocking shadow mode, then make an evidence-based decision about targeted automatic Copilot review requests.

## Prerequisites

- Skill structure/content is available from the authoring sub-issue.
- Offline benchmark results meet or clearly characterize the parent issue's promotion metrics.

## Deliverables

- [ ] Request Copilot review manually on at least 10 PRs changing `tools/azsdk-cli/**`.
- [ ] Score each comment as accepted, rejected, duplicate, or unclear with a reason.
- [ ] Inspect comment attributions to confirm whether the skill or MCP context contributed.
- [ ] Inspect linked review-session logs to record files, skills, MCP tools, and context used.
- [ ] Add confirmed misses and false positives to the frozen fixture corpus and rerun it.
- [ ] Publish shadow precision, evidence, duplicate, scope, and stability results.
- [ ] Decide whether promotion criteria justify automated reviewer requests.
- [ ] If promoted, implement an idempotent path-filtered workflow that requests `copilot-pull-request-reviewer[bot]` only for `tools/azsdk-cli/**` changes.
- [ ] Keep reviews report-only and non-blocking.

## Guardrails

GitHub's native automatic-review ruleset targets branches rather than repository paths. Do not enable repository-wide automatic review as the initial experiment. Prefer manual shadow requests, then a narrowly path-filtered workflow if the measured results support promotion.

## Acceptance criteria

- At least 10 real PRs have a complete, auditable scorecard.
- Skill attribution and session usage are recorded where available.
- The promotion decision cites measured results and known failure modes.
- Any automation is path-scoped, idempotent, non-blocking, and documented with a rollback path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.