Tracking Issue for compiletest documentation

Context

compiletest is the main test harness of rustc and other tools inside this repository, but there is a significant lack of documentation for it. We should write some documentation for compiletest. There are some documentation in rustc-dev-guide, but compiletest is a lot more complex than that.

Areas lacking documentation

• High-level architecture and overview
• Interaction between bootstrap <-> compiletest <-> tests
• compiletest-as-a-binary: CLI interface options and docs
• Filtering tests
• How are test modes and test suites created, registered, built and run
  • When can tests be ignored? cached test results and their invalidation: dependency stamps
  • Some test mode / test suite tests can be run in parallel / concurrently:
    • Concurrency assumptions: what if the test modifies external state?
• Test mode and test suite specific behavior, assumptions and quirks
  • UI tests: auxiliary crates and bins, compile-flags, compare modes, assumptions and conventions on stdout/stderr output, blessing
  • run-make and run-make-fulldeps tests: how the whole build pipeline is setup, concepts, how tests are built and executed, provided env vars, assumed dependencies
  • codegen tests: filecheck, revisions magic behavior
  • coverage tests: how to bless?
  • ui-fulldeps have some non-trivial behavior: e.g. https://github.com/rust-lang/rust/pull/126073#discussion_r1629843686
• Error handling strategy/design and diagnostics
• Interaction with CI
• Test directive collection, parsing and validation
  • Which directives can be used in which test suites?
  • What do the directives even mean/do?
• Test output normalization special syntax
  • e.g. const CWD: &str = "{{cwd}}";
• Setting current_dir when running tests

Related issues

• https://github.com/rust-lang/rust/issues/125585
• https://github.com/rust-lang/rust/issues/125587
• https://github.com/rust-lang/rust/issues/126092

Unresolved questions

• Where do we keep compiletest docs?
  • Maybe alongside the source? Maybe a mdBook? rustc-dev-guide?
  • Ideally we would like this easy to read and reference and improve.
• How to we keep compiletest documentation update-to-date?
  • Periodic triage?

Documentation improvement history

TODO

Start with the existing rustc-dev-guide compiletest documentation and the related issues #125585, #125587, and #126092. Determine where the documentation should live and which listed areas to cover; done means a maintainable, readable compiletest reference addressing an agreed scope and update process.