rust-lang / rust-lang/rust

Audit {`check-run-results`, `regex-error-pattern`, `error-pattern`, `check-stdout`, `normalize-*`, `dont-check-compiler-*`, `run-rustfix`, `rustfix-only-machine-applicable`, `forbid-output`} family of directives

Open
#134,888 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A-compiletest A-testsuite C-bug E-hard E-needs-design E-needs-investigation T-bootstrap T-compiler
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

I don't know about other contributors, but I have to look at the source implementation (or trial-and-error) every time I see or try to use one of these compiletest directives. IMO, we should audit the design of these directives, and possibly revamp them entirely.

Concrete confusions:

  • check-run-results check both run stderr and stdout and puts them into snapshot files (on bless), then compares the subsequent run stderr and stdout against the snapshot.
  • error-pattern doesn't only check stderr (whose stderr? compiler? run?), it can also check stdout (or both??) depending on check-stdout, dont-check-compiler-*, and also it can check also compiler stderr or stdout I think??
  • normalize-* (that is not normalize-stdout or normalize-stderr) I believe can simultaneously apply to {compiler,run} {stderr,stdout}.
  • run-rustfix will run rustfix and try to apply all non-placeholder suggestions, including non-machine-applicable ones like MaybeIncorrect ones.
  • rustfix-only-machine-applicable is like run-rustfix but only tries to apply MachineApplicable suggestions.
  • forbid-output is like error-pattern but named completely differently. I don't remember which output pattern of {compiler,run}x{stderr,stdout} it is forbidding.

EDIT:

  • regex-error-pattern is like error-pattern but accepts a regex...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the compiletest directive implementations for check-run-results, error-pattern, regex-error-pattern, check-stdout, normalize-, dont-check-compiler-, run-rustfix, rustfix-only-machine-applicable, and forbid-output. Trace how each directive selects compiler or run output and how snapshots or rustfix are handled. Done means the directive behavior and naming are audited and the issue's listed ambiguities have a decided, documented resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.