Azure / Azure/azure-sdk-tools

Improve CCR quality across all SDK repos

Open
#16,034 3 comments 2 reactions 2 assignees Claimed by @richardpark-msft View on GitHub
dev inner loop
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

## Goal

- Reduce human burden for code reviews by improving reliability of CCR
- Improve CCR quality by automatically encoding repeated missed patterns in prompts in a repeated agentic workflow
- Track quantitative metrics to measure progress

## Solution

Standardize organization of code review prompts across repos for CCR to leverage.

Periodically mine recently closed/merged PRs, separate Copilot-reviewer comments from human comments, use an LLM judge to find substantive, issues that humans caught but Copilot missed, cluster those into themes, and track metrics over time. Based on themes and patterns, suggests generalizable prompt improvements.

## Implementation overview (high-level)

### Reviewer skill

- A universal reviewer skill lives in and syncs from `azure-sdk-tools`. It points towards language-specific skills
- Each repo has its own specific rules in its own skill file
- Enforce sub-agent usage for different focus areas, e.g. security review, architecture review, etc.

### Automated prompt enhancement workflow

1. Ingest merged PR data from a custom date range.
2. Split bot/Copilot comments vs. human comments. Key idea: consider what humans had to catch that Copilot missed
3. Filter low-signal noise, nitpicky comments, only consider comments from core contributors, etc.
4. Cluster comments into themes/patterns
5. Normalize themes to avoid overfitting (does this missed pattern occur in multiple PRs?)
6. Extract and report metrics
7. Open PR/issue with suggested prompt updates

## Open questions / ideas

- Some code review patterns are specific to codepaths or packages, not generalizable to the entire repo. Should we attempt to automate suggestions to package-specific review prompts?
- What metrics do we track? e.g. # human comments over time, # human-caught patterns per 'category' over time (security, bug, architecture) that Copilot missed, time it takes to merge a PR
- How do we aggregate/present the metrics?
- What PR data are we considering when analyzing comments? e.g. should we consider if a comment was acted on (a later commit touches the file line range covered by that comment),
- Should we also consider Copilot comments that were not acted upon / had negative responses? aka false positives?
- Do we want a DB to store run information ? maybe not, can chain agentic workflows and look at outputs?
- Should we consider bug fixes we had to merge, and investigate at root cause and why that wasn't caught in PR review?

## Related references:
- https://github.com/Azure/azure-sdk-for-python/pull/47506
- https://github.com/dotnet/fsharp/blob/822e80493c45d55706357c5b6670295e99f77eec/.github/agents/extraction-pipeline.md

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.