Tracking Issue for enforcing test doc comments

Please use the t-compiler zulip thread for specific discussions or open new issues, but please feel free to post consensus updates or progress updates on this tracking issue.

Zulip thread: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Better.20document.20the.20intent.20of.20a.20test.3F

Context

Way too often, tests get added, but they don't have any included context in the form of comments. This includes things like (or rather, the lack of important information such as):

• What the test is trying to check.
• Test context:
  • Related issues (issue number for regression tests),
  • Previous implementation PRs
  • RFC / Rust Reference / external docs links (syscalls, platform APIs, DWARF standards, ISA docs, you name it)
  • When suitable, how/why the regression occurred / was reachable, how it was fixed, and how does the test exercise the implementation such that it will catch the same regression.

A good litmus test for test doc comments is: 5 years later, will I be able to determine what the test is trying to check without having to jump through a bunch of github issues/PRs via git blame?

So, it might be valuable to add an automated check to enforce that every new test is checked in with a suitable test doc comment.

Implementation steps

• 1. Briefly survey existing test suites and see what kind of doc comment style are suitable.
  • Notably, which test suites should / should not have doc comments enforced, e.g. rustdoc-gui and other special test suites may need special handling or exclusions.
  • Figure out a brief mechanism such that new tests receive the enforcement but old tests are permitted in an allowlist, and how to bless that allowlist.
• 2. File an MCP describing the previous rationale for enforcing test doc comments, as well as the concrete proposal for how it will be implemented and the proposed UX. This may need to be encoded as T-compiler PR/review policy so may need a full team FCP.
• 3. Implement the test doc comment enforcement mechanism, including sufficient self-test coverage.
• 4. Update rustc-dev-guide ui test walkthrough + basic test description + best practices.
• 5. Brief T-compiler/T-rustdoc teams about the new test doc comment enforcement.

Discussions

• https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Better.20document.20the.20intent.20of.20a.20test.3F

Unresolved questions / concerns

None yet, please update issue description as suitable.

Start by surveying the existing Rust test suites, with special attention to rustdoc-gui and other suites that may need exclusions, and review the linked Zulip discussion. Then define the allowlist and proposed enforcement UX before implementing the checker with self-tests. Done means new tests are enforced, existing tests remain handled by the allowlist, and the rustc-dev-guide walkthrough and best practices are updated.