Azure / Azure/azure-rest-api-specs
ARM API Reviewer agent: define label precedence and human escalation for auto-signed-off PRs
- Dominant language
- TypeSpec
- Stars
- 3.1k
- Forks
- 5.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 424
Description
The ARM API Reviewer can uncover a new Blocking concern on a PR that has already received automated ARM sign-off. Today, automation can remove the resulting `ARMChangesRequested` label, leaving the PR labels indicating sign-off while the concern remains unresolved. We need a clear precedence and human-escalation policy so automated sign-off and reviewer decisions remain consistent.
### Background
The ARM API Reviewer can identify a new Blocking issue after automated ARM sign-off has already been applied or while the auto-signoff workflow is still running.
The current label state machine does not define precedence between these signals:
- Auto-signoff adds `ARMSignedOff` and an `ARMAutoSignedOff-*` label.
- An ARM reviewer may add `ARMChangesRequested` after validating a new Blocking finding identified by the ARM API Reviewer agent.
- The auto-signoff and summarize-checks workflows can then remove `ARMChangesRequested` because `ARMSignedOff` takes precedence.
As a result, a reviewer-selected state can disappear while the Blocking finding remains unresolved.
The proposed operating model is to exclude automatically signed-off PRs from the routine manual review queue while still escalating serious new issues identified by the ARM API Reviewer agent for human judgment. This issue tracks the policy decisions, label state machine, queue behavior, and documentation needed to implement that model.
Warnings, Suggestions, and Existing Blocking findings are not part of this escalation scenario.
### Impact evidence
Telemetry from June 11 through August 11, 2026 identified 14 override episodes across 12 PRs where:
1. The same review session posted at least one finding marked `severity: blocking` and `classification: new`.
2. The reviewer added `ARMChangesRequested`.
3. Automation subsequently removed the reviewer-selected state.
Ten of the 14 reversals occurred within two minutes. The median reversal time was 72 seconds. One PR's findings were later corrected from Blocking to Warning, leaving a conservative cohort of 13 episodes across 11 PRs.
Public examples include:
- [PR #44536](https://github.com/Azure/azure-rest-api-specs/pull/44536#discussion_r3641177258): the reviewer reported that a TypeSpec property was missing from the generated OpenAPI. `ARMChangesRequested` was removed twice by automation.
- [PR #45361](https://github.com/Azure/azure-rest-api-specs/pull/45361#discussion_r3751180738): the reviewer reported collection data modeled as a comma-separated string. `ARMChangesRequested` was removed 56 seconds after it was added.
- [PR #44716](https://github.com/Azure/azure-rest-api-specs/pull/44716): two reviewer decisions were reversed while automated sign-off was present. The findings were later discussed and resolved by the reviewer.
- [PR #44941](https://github.com/Azure/azure-rest-api-specs/pull/44941#discussion_r3647833902): the initial Blocking classification was later corrected to Warning. This demonstrates why some findings identified through AI-assisted review require human judgment rather than automatically replacing sign-off.
Four affected PRs later merged after additional review activity. The evidence does not show that any PR merged immediately because of the override, but the current behavior can hide the intended review state and make the queue and labels unreliable.
### Decisions required
1. **Reviewer agent authority:** Should a critic-validated NEW Blocking finding remove `ARMSignedOff` and its `ARMAutoSignedOff-*` companion labels, or should removing sign-off remain a human-only action?
2. **Auto-signoff precedence:** Should auto-signoff treat `ARMChangesRequested`, removal of an auto-signoff label, or a new escalation label as a veto? What event should allow auto-signoff to run again?
3. **Human escalation label:** Should the workflow add a new label such as `WaitForHuman`, or reuse `WaitForARMFeedback`, when it posts a NEW Blocking finding that needs human judgment?
4. **Escalation scope:** Should the reviewer continue evaluating ARM, TypeSpec, RPC, and general API guidance while reserving the escalation label strictly for NEW Blocking findings?
5. **Queue behavior:** Should automatically signed-off PRs be excluded from the normal ARM review queue unless an escalation label is present?
6. **Resolution authority:** Who may clear the escalation state, and what evidence is required when a finding is fixed, downgraded, or determined not to apply?
7. **Merge behavior:** Should the escalation label be informational, affect only the ARM review queue, or block merging until a human records a decision?
### Acceptance criteria
- Document an explicit state-transition table covering manual sign-off, auto-signoff, `ARMChangesRequested`, and the selected human-escalation label.
- Define label ownership, precedence, and the conditions under which each label can be added, removed, or restored.
- Cover both observed race conditions: auto-signoff already present before review and auto-signoff completing after review.
- Confirm that Warning, Suggestion, and Existing Blocking findings do not change sign-off or trigger human escalation.
- Update the ARM reviewer queue filter and on-call expectations to exclude auto-signed-off PRs unless escalation is required.
- Add tests for label transitions, race conditions, finding downgrades, and human resolution.
- Update the ARM API Reviewer and workflow documentation with the selected behavior.
- Preserve enough telemetry to distinguish automated sign-off, reviewer-agent escalation, and human decisions.
### Related work
- [#45350](https://github.com/Azure/azure-rest-api-specs/issues/45350) tracks ARM API Reviewer workflow reliability and reconciliation.
- [PR #45351](https://github.com/Azure/azure-rest-api-specs/pull/45351) implements the associated workflow hardening.
- [#43350](https://github.com/Azure/azure-rest-api-specs/issues/43350) introduced the automated review trigger.
Contributor guide
Research direction
Start by reviewing related issue #45350 and PR #45351, then trace the ARM API Reviewer workflows and label transitions described here. Define and document the state-transition policy, queue behavior, escalation ownership, telemetry, and race-condition handling; done requires the specified transition tests and documentation updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- devops, documentation, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100