[azsdk-cli code review] Run a shadow-review pilot and decide auto-review
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
## Parent
- #16764
## Goal
Validate the skill on real azsdk-cli pull requests in non-blocking shadow mode, then make an evidence-based decision about targeted automatic Copilot review requests.
## Prerequisites
- Skill structure/content is available from the authoring sub-issue.
- Offline benchmark results meet or clearly characterize the parent issue's promotion metrics.
## Deliverables
- [ ] Request Copilot review manually on at least 10 PRs changing `tools/azsdk-cli/**`.
- [ ] Score each comment as accepted, rejected, duplicate, or unclear with a reason.
- [ ] Inspect comment attributions to confirm whether the skill or MCP context contributed.
- [ ] Inspect linked review-session logs to record files, skills, MCP tools, and context used.
- [ ] Add confirmed misses and false positives to the frozen fixture corpus and rerun it.
- [ ] Publish shadow precision, evidence, duplicate, scope, and stability results.
- [ ] Decide whether promotion criteria justify automated reviewer requests.
- [ ] If promoted, implement an idempotent path-filtered workflow that requests `copilot-pull-request-reviewer[bot]` only for `tools/azsdk-cli/**` changes.
- [ ] Keep reviews report-only and non-blocking.
## Guardrails
GitHub's native automatic-review ruleset targets branches rather than repository paths. Do not enable repository-wide automatic review as the initial experiment. Prefer manual shadow requests, then a narrowly path-filtered workflow if the measured results support promotion.
## Acceptance criteria
- At least 10 real PRs have a complete, auditable scorecard.
- Skill attribution and session usage are recorded where available.
- The promotion decision cites measured results and known failure modes.
- Any automation is path-scoped, idempotent, non-blocking, and documented with a rollback path.
Contributor guide
Assessment
This issue has not been assessed yet.