Azure / Azure/azure-rest-api-specs
Evaluate and enable Copilot review of API documentation accuracy
- Dominant language
- TypeSpec
- Stars
- 3.1k
- Forks
- 5.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 424
Description
## Goal
Evaluate and enable explicit Copilot review of customer-facing API documentation in `azure-rest-api-specs` pull requests so inaccurate descriptions are caught before they propagate into generated SDK comments, REST reference documentation, and samples.
## Current state
The repository already has three related mechanisms:
- GitHub Copilot Code Review instructions in `.github/copilot-review-instructions.md`, but no repository ruleset automatically requests this review; it is currently used only when explicitly requested.
- The Copilot-powered ARM API Reviewer workflow, triggered through the ARM review process.
- The label-triggered, advisory data-plane API Reviewer workflow.
Existing explicit Copilot reviews have found valuable documentation defects, including supported/unsupported wording copied incorrectly, PATCH omission described as applying a default, response text inconsistent with `200` versus `204`, and shared TypeSpec documentation unintentionally changing older API versions. However, generic Copilot review can overlap with the custom API reviewers and can duplicate findings on TypeSpec source and generated artifacts.
## Review requirements
Review the authoritative TypeSpec source:
- Review `.tsp` doc comments and `@doc` content.
- Use generated artifacts only to verify the result, and report the fix against TypeSpec rather than duplicating findings on generated files.
- Review source examples because they are published and used by downstream SDK/docs tooling.
Prioritize substantive accuracy and completeness over grammar preferences. Check that documentation agrees with:
- names, types, required/optional status, visibility, defaults, units, constraints, and secret handling;
- operation semantics, status codes, paging, LRO behavior, errors, and deprecation guidance;
- examples and previous API versions; and
- the scope/version in which a TypeSpec feature is introduced.
Apply existing style guidance: meaningful descriptions, customer-facing terminology, correct acronym casing, units where applicable, no descriptions that merely repeat a name, and idempotent PUT wording.
For service behavior not verifiable from the TypeSpec, examples, or repository evidence, ask for confirmation rather than asserting that the documentation is wrong.
## Possible approaches
### A. Automatically request GitHub Copilot Code Review
Add `copilot_code_review` to the repository ruleset and use the existing `.github/copilot-review-instructions.md`.
- **Pros:** simple, consistent, and produces standard inline Copilot review comments.
- **Cons:** may duplicate ARM/data-plane reviewer or linter findings and offers less workflow control.
### B. Extend the existing ARM and data-plane reviewers
Add a shared documentation-accuracy rubric to the current agent workflows and their evaluation suites.
- **Pros:** reuses existing scope, critic, safety, reconciliation, and evaluation infrastructure and avoids a third reviewer.
- **Cons:** only runs when those workflows are triggered and may not provide the explicitly requested GitHub Copilot Code Review experience.
### C. Pilot both, then choose
Run explicit Copilot Code Review on a controlled set of ARM TypeSpec, data-plane TypeSpec, and documentation-only PRs. Compare its unique findings and noise with the custom reviewers, then choose automatic, conditional, or agent-only review.
This is the recommended starting approach.
## Work
1. Add a shared documentation-quality reference that can be consumed by generic Copilot Code Review and the custom API reviewers.
2. Update review instructions to enforce TypeSpec source ownership, accuracy checks, generated-file deduplication, evidence requirements, and low-noise severity/comment limits.
3. Add evaluations based on historical defects and true negatives, covering TypeSpec source versus generated output, cross-version leakage, status-code semantics, defaults, units, required/optional mismatches, example consistency, and unverifiable service claims.
4. Pilot explicit Copilot reviews and measure actionable findings, false positives, duplicates with linters/reviewers, and comments incorrectly targeting generated files.
5. Decide and document the long-term trigger model:
- automatic repository ruleset;
- conditional request for relevant TypeSpec changes;
- description/example-focused review; or
- custom ARM/data-plane reviewers only.
6. Keep the review advisory during evaluation. Do not count Copilot as a required approval until accuracy and operating policy are agreed.
## Done when
- A shared, evidence-based documentation review rubric covers TypeSpec comments, examples, and version consistency.
- Generated output does not receive duplicate or unactionable findings.
- Historical positive cases and clean negative cases are evaluated.
- Pilot results quantify precision and overlap with existing checks.
- The team explicitly selects and enables, or rejects, a long-term review approach.
- Contributor guidance explains the review's scope, advisory status, and how to handle incorrect findings.
- by copilot
Contributor guide
Assessment
This issue has not been assessed yet.