Azure / Azure/azure-rest-api-specs

Evaluate and enable Copilot review of API documentation accuracy

Open
#46,129 0 comments 0 reactions 2 assignees Claimed by @XiaofeiCao View on GitHub
documentation Tooling Team
Dominant language
TypeSpec
Stars
3.1k
Forks
5.9k
Avg merge
3d 2h
Merged PRs (30d)
424

Description

## Goal

Evaluate and enable explicit Copilot review of customer-facing API documentation in `azure-rest-api-specs` pull requests so inaccurate descriptions are caught before they propagate into generated SDK comments, REST reference documentation, and samples.

## Current state

The repository already has three related mechanisms:

- GitHub Copilot Code Review instructions in `.github/copilot-review-instructions.md`, but no repository ruleset automatically requests this review; it is currently used only when explicitly requested.
- The Copilot-powered ARM API Reviewer workflow, triggered through the ARM review process.
- The label-triggered, advisory data-plane API Reviewer workflow.

Existing explicit Copilot reviews have found valuable documentation defects, including supported/unsupported wording copied incorrectly, PATCH omission described as applying a default, response text inconsistent with `200` versus `204`, and shared TypeSpec documentation unintentionally changing older API versions. However, generic Copilot review can overlap with the custom API reviewers and can duplicate findings on TypeSpec source and generated artifacts.

## Review requirements

Review the authoritative TypeSpec source:

- Review `.tsp` doc comments and `@doc` content.
- Use generated artifacts only to verify the result, and report the fix against TypeSpec rather than duplicating findings on generated files.
- Review source examples because they are published and used by downstream SDK/docs tooling.

Prioritize substantive accuracy and completeness over grammar preferences. Check that documentation agrees with:

- names, types, required/optional status, visibility, defaults, units, constraints, and secret handling;
- operation semantics, status codes, paging, LRO behavior, errors, and deprecation guidance;
- examples and previous API versions; and
- the scope/version in which a TypeSpec feature is introduced.

Apply existing style guidance: meaningful descriptions, customer-facing terminology, correct acronym casing, units where applicable, no descriptions that merely repeat a name, and idempotent PUT wording.

For service behavior not verifiable from the TypeSpec, examples, or repository evidence, ask for confirmation rather than asserting that the documentation is wrong.

## Possible approaches

### A. Automatically request GitHub Copilot Code Review

Add `copilot_code_review` to the repository ruleset and use the existing `.github/copilot-review-instructions.md`.

- **Pros:** simple, consistent, and produces standard inline Copilot review comments.
- **Cons:** may duplicate ARM/data-plane reviewer or linter findings and offers less workflow control.

### B. Extend the existing ARM and data-plane reviewers

Add a shared documentation-accuracy rubric to the current agent workflows and their evaluation suites.

- **Pros:** reuses existing scope, critic, safety, reconciliation, and evaluation infrastructure and avoids a third reviewer.
- **Cons:** only runs when those workflows are triggered and may not provide the explicitly requested GitHub Copilot Code Review experience.

### C. Pilot both, then choose

Run explicit Copilot Code Review on a controlled set of ARM TypeSpec, data-plane TypeSpec, and documentation-only PRs. Compare its unique findings and noise with the custom reviewers, then choose automatic, conditional, or agent-only review.

This is the recommended starting approach.

## Work

1. Add a shared documentation-quality reference that can be consumed by generic Copilot Code Review and the custom API reviewers.
2. Update review instructions to enforce TypeSpec source ownership, accuracy checks, generated-file deduplication, evidence requirements, and low-noise severity/comment limits.
3. Add evaluations based on historical defects and true negatives, covering TypeSpec source versus generated output, cross-version leakage, status-code semantics, defaults, units, required/optional mismatches, example consistency, and unverifiable service claims.
4. Pilot explicit Copilot reviews and measure actionable findings, false positives, duplicates with linters/reviewers, and comments incorrectly targeting generated files.
5. Decide and document the long-term trigger model:
- automatic repository ruleset;
- conditional request for relevant TypeSpec changes;
- description/example-focused review; or
- custom ARM/data-plane reviewers only.
6. Keep the review advisory during evaluation. Do not count Copilot as a required approval until accuracy and operating policy are agreed.

## Done when

- A shared, evidence-based documentation review rubric covers TypeSpec comments, examples, and version consistency.
- Generated output does not receive duplicate or unactionable findings.
- Historical positive cases and clean negative cases are evaluated.
- Pilot results quantify precision and overlap with existing checks.
- The team explicitly selects and enables, or rejects, a long-term review approach.
- Contributor guidance explains the review's scope, advisory status, and how to handle incorrect findings.

- by copilot

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.