microsoft / microsoft/GitHub-Copilot-for-Azure

Copilot PR reviewer creates infinite fix→review→fix cycle on skill PRs

Open
#2,039 2 comments 0 reactions 1 assignee Claimed by @kvenkatrajan View on GitHub
in-progress infrastructure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Problem

Every push to a skill PR triggers `copilot-pull-request-reviewer`, which finds new nitpicks. We fix them, push, and the reviewer fires again with more comments. This creates an unbreakable whack-a-mole cycle that blocks PRs from getting clean.

**Example from PR #1636:**
- Started with 24 review threads (copilot-reviewer + JonG)
- Fixed all, pushed → 6 new copilot comments appeared
- Fixed those, pushed → 7 MORE new copilot comments appeared
- Comments are increasingly stylistic/pedantic (e.g. "variable name not defined in snippet" on reference docs that aren't meant to be compilable)

Same pattern on #1637 and #1642.

## Root Cause

The copilot-reviewer treats **reference documentation** (markdown with embedded Bicep snippets) the same as **production code**. It flags things like:
- Undefined variables in illustrative snippets (`uamiClientId` referenced but not declared as param)
- Design choices as bugs (`minReplicas: 0` vs `1`)
- Wording precision ("copy" vs "create" for recipe modules)
- PR description not matching issue deliverables exactly

These are valid observations for **code review** but inappropriate strictness for **reference documentation templates** that are intentionally simplified.

## Impact

- PRs #1636, #1637, #1642 have been through 3+ fix cycles each
- Each cycle takes ~15-30 min (read comments, fix, push, wait for CI, resolve threads)
- Human reviewers (JonG, Simon) already approved the substance — copilot keeps finding new surface-level issues

## Proposed Solutions

1. **Configure copilot-reviewer to be less strict on `references/` and `templates/` paths** — these are docs, not code. Lower the sensitivity or skip certain categories of findings.
2. **Add a `copilot-review: skip` label or comment** that suppresses re-review on follow-up pushes after human approval
3. **Scope the reviewer to only review the initial diff**, not re-review the entire file on every push
4. **Separate code-quality checks from doc-quality checks** — different rules for `.md` files vs `.ts`/`.bicep` files

## Current Workaround

We're triaging comments into "fix" vs "reply-and-resolve as intentional", then batch-pushing + immediately resolving all threads via GraphQL API before the next review cycle can pile up. This works but is manual and fragile.

cc @paulyuk @simonjj

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.