Azure / Azure/azure-sdk-tools

Run REX documentation validation in public PR pipelines

Open
#16,933 1 comment 0 reactions 2 assignees Claimed by @XiaofeiCao View on GitHub
needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

## Goal

Run REX documentation generation against packages built by public pull-request pipelines so malformed API doc comments are caught before nightly docs builds or release.

REX is already used during docs publishing:

- Java: `java2docfx`
- Python: `py2docfx`
- JavaScript: `@microsoft/type2docfx`

The tools are public, but current validation is coupled to `Update-DocsMsMetadata.ps1` and normally downloads an already-published package. PR validation must instead use the package artifacts produced by the current build.

## Work

1. **Extract reusable validation orchestration in `eng/common`**
- Add a script/pipeline template that reads `PackageInfo` files, invokes the language-specific `Validate--DocMsPackages` implementation, aggregates failures, and fails with the affected package names.
- Keep docs-repository checkout and metadata updates out of this path.
- Reuse the same underlying validation from release/docs publishing to prevent behavior drift.

2. **Support local package artifacts in each language repository**
- **Java:** run `java2docfx` with package coordinates and the staged local Maven package folder.
- **Python:** configure `py2docfx` with a local wheel/source distribution (`dist_file`) and make other PR-built wheels available for dependencies.
- **JavaScript:** pass the staged `.tgz` directly to `type2docfx`, ensuring packages changed together can resolve one another.

3. **Add public PR validation jobs**
- Java: depend on `Build`, download `packages`.
- Python: depend on `Build_Extended`, download `packages_extended`.
- JavaScript: depend on `Build`, download `packages`.
- Discover affected packages from generated `PackageInfo`, since `Service=auto` determines the package set dynamically.
- Match existing docs eligibility rules, including `skipPublishDocMs`, private/non-publishable packages, and Java implementation-package exclusions.
- Missing expected artifacts should fail as infrastructure errors; no eligible packages should be a successful no-op.

4. **Test and roll out**
- Add tests for local-artifact selection, multiple-package aggregation, skipped packages, no-op behavior, and missing artifacts.
- Use public test PRs per language to prove a valid package passes, a malformed doc comment fails, and fixing only the comment passes.
- Start non-blocking to identify false positives and measure duration, then make the check required.
- Provide a pipeline-variable kill switch for rapid rollback.

## Done when

Public Java, Python, and JavaScript pull-request builds run REX against the exact packages built by the PR, report package-specific failures, require no internal feed credentials, and block malformed documentation before merge.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.