compiletest: Deficiencies and pitfalls of `//@ normalize-*` headers
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 119k
- Forks
- 16.1k
- PR merge metrics
- PR metrics pending
Description
These headers allow individual tests to define custom output-normalization rules, so that output snapshots are less sensitive to things like platform differences, line numbers, and timestamps.
Unfortunately, they have a few limitations and pitfalls:
- The value parser for these headers is very lax, and when it does fail to parse, the header is silently ignored.
- #126370
- Other headers have similar problems: #123760
- The parser doesn't require a colon
:after the header name, so the test suite contains an inconsistent mix of with-colon and without-colon rules.- #126777
- The rules cannot contain literal double-quote
"characters, which is very awkward for some kinds of output (e.g. JSON). - The parser uses
parse_cfg_name_directiveto help read the header name, which is sometimes useful (e.g. for defining separate rules for 32-bit and 64-bit), but most of the time it means that tests have to awkwardly writenormalize-stderr-testinstead of the more intuitivenormalize-stderr. - The rules are applied to both compile output and run output, which is usually not a problem in practice, but can be unexpected.
- Like many other headers, they are only meaningful in
ui-mode tests, and are silently ignored if they appear in other kinds of test.
@rustbot label +A-compiletest
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue concerns compiletest's //@ normalize-* header parsing and lists several limitations, with some items already linked to #126370, #126777, and #123760. Start by locating the parser and the UI-mode tests that use these headers, then select an unchecked limitation, define its expected behavior, and add regression coverage showing that the chosen issue is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100