Azure / Azure/azure-rest-api-specs
ARM API Reviewer agent: enforce and verify a visible attribution preamble on every posted review
- Dominant language
- TypeSpec
- Stars
- 3.1k
- Forks
- 5.9k
- Avg merge
- 3d 37m
- Merged PRs (30d)
- 446
Description
### Background
The [ARM API Reviewer agent](https://github.com/Azure/azure-rest-api-specs/blob/main/.github/agents/arm-api-reviewer.agent.md) posts its findings as inline PR review comments. Step 8 of the agent definition specifies a "Review-body preamble", a visible `## ARM API Review` block stating that the comments come from the agent, that they are critic-verified, and against which commit. Separately, every posted comment ends with a hidden HTML marker ``.
In practice the visible preamble is frequently missing. When the agent posts each finding as its own single-comment review with an empty review body, the only provenance left on the PR is the hidden HTML marker, which does not render in Markdown. To a reader, the findings then look like unattributed, context-free comments posted under the reviewer personal GitHub handle.
A spec author hit this on PR #44941: they could not tell where the comments came from and assumed they were spurious. The root cause is that Step 8 describes the preamble but does not (a) require posting through a single review that carries the preamble as its body, and (b) verify after posting that a visible preamble actually landed.
### Evidence
Trigger case, PR #44941:
- The original run (2026-07-24, operator `sandipsh`) submitted two `COMMENTED` reviews with empty bodies (`bodyLen=0`), so no visible preamble was posted. Affected comments: [r3647833902](https://github.com/Azure/azure-rest-api-specs/pull/44941#discussion_r3647833902), [r3647834008](https://github.com/Azure/azure-rest-api-specs/pull/44941#discussion_r3647834008).
- A later run (2026-07-27, operator `ravimeda`) correctly added a review whose body begins with `## ARM API Review`. This shows the intended behavior is well defined but not enforced, so whether attribution appears depends on the run.
Prevalence. Of 42 PRs sampled that carry the agent hidden marker, 18 had no visible attribution anywhere (review body or top-level comment) when first checked, and 17 still have none as of this writing (PR #44941 was remediated by the later run). The gap spans more than one operator, so it is systemic to the workflow rather than a single mistake.
Currently missing visible attribution:
- Operator `sandipsh`: #44899, #44830, #44807, #44803, #44796, #44769, #44745, #44376, #43926, #43925, #43898, #43894, #43814, #43745, #43694
- Operator `razvanbadea-msft`: #44750, #44330
### Related observation (track separately)
On PR #44941 the two original comments are also textually corrupted (for example `^Gpi-version` in place of `api-version`, and stray backslashes where `$filter` and `$top` should appear). This is a posting-time escaping defect, where the comment body was built by shell string interpolation instead of JSON serialization. It shares the same Step 8 posting area but is a distinct defect from the missing attribution and can be fixed alongside it.
### References
- Agent definition, Step 8 "Review-body preamble": [.github/agents/arm-api-reviewer.agent.md](https://github.com/Azure/azure-rest-api-specs/blob/main/.github/agents/arm-api-reviewer.agent.md)
- Reviewer and Critic protocol (telemetry marker vs visible attribution): [.github/agents/protocols/arm-api-review-critic.protocol.md](https://github.com/Azure/azure-rest-api-specs/blob/main/.github/agents/protocols/arm-api-review-critic.protocol.md)
- User docs: [documentation/api-reviewer-agent.md](https://github.com/Azure/azure-rest-api-specs/blob/main/documentation/api-reviewer-agent.md)
- Eval harness: [.github/skills/evals/arm-api-reviewer/](https://github.com/Azure/azure-rest-api-specs/tree/main/.github/skills/evals/arm-api-reviewer)
- Trigger PR: #44941
### Requested work
1. Make the visible preamble mandatory in Step 8. Post all inline findings from a run through a single review (`create_pull_request_review`) whose body is the `## ARM API Review` preamble. Do not post findings as separate empty-body reviews.
2. Add a fallback. When the posting path cannot set a review body, post the preamble as a top-level PR comment so visible attribution always exists.
3. Add a post-condition check. After posting, re-fetch the review and top-level comments and assert that at least one visible body contains the preamble. If none does, post the fallback top-level comment before the run finishes.
4. Consider a short visible attribution line on each inline comment, not only the hidden marker, so a stray single comment still carries provenance.
5. Add eval coverage. Add a check to the eval harness that fails when a simulated posting run omits the visible preamble.
6. Backfill (optional but recommended). Run a one-time reconciliation that adds a preamble or a top-level attribution comment to the open PRs listed under Evidence.
### Acceptance criteria
- Every run that posts at least one finding also posts a visible `## ARM API Review` preamble (review body or top-level comment), confirmed by a re-fetch post-condition.
- No run leaves findings whose only provenance is the hidden HTML marker.
- The eval harness has a check that fails when a posting run omits the visible preamble.
- The open PRs listed under Evidence either have a visible preamble added or are documented as remediated.
Contributor guide
Research direction
Start with Step 8 in .github/agents/arm-api-reviewer.agent.md and compare its posting flow with arm-api-review-critic.protocol.md. Inspect .github/skills/evals/arm-api-reviewer/ and the create_pull_request_review path, then verify that a run with findings leaves a visible ## ARM API Review preamble after re-fetching reviews and top-level comments; the eval should fail when it does not.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, markdown, shell
- Domain
- testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100