compiletest should note when test output is normalized
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 119k
- Forks
- 16.1k
- PR merge metrics
- PR metrics pending
Description
compiletest has two basic kinds of normalization:
- normalization it does itself, either built-in or through a
normalize-stderrpattern. this is what the UI suite uses. - normalization it doesn't know about, e.g. in a run-make test
normalizing outputs can lead to some extremely confusing errors, such as https://github.com/rust-lang/rust/pull/121571#issuecomment-2129928216. to avoid misleading contributors, it should note when the output has been normalized.
the first part is not too hard; it can save list of lines that have been modified in each test and print something like "note: 32/32 lines that did not match were normalized before comparison; see build/test/ui/foo.out.orig for unnormalized version"
(note that the unnormalized file already exists today but almost nobody knows about it).
the second part is trickier; run-make tests often run sed before comparing, and this is opaque to compiletest. perhaps fixing this can be part of the ongoing work to switch run-make to rust? and, in the meantime, introduce a new SED script, based off cut-and-grep.sh, to prevent the problem from getting worse? cc @jieyouxu
@rustbot label A-contributor-roadblock A-diagnostics D-confusing A-testsuite
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the compiletest normalization flow and the existing unnormalized output at build/test/ui/foo.out.orig. Compare how built-in and normalize-stderr changes are tracked, then inspect cut-and-grep.sh and run-make handling. Done means normalized output is clearly reported with a path to the unnormalized version, without misleading comparison errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100