rust-lang / rust-lang/rust

compiletest: Deficiencies and pitfalls of `//@ normalize-*` headers

Open
#126,372 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A-compiletest A-testsuite C-bug E-medium T-bootstrap
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

These headers allow individual tests to define custom output-normalization rules, so that output snapshots are less sensitive to things like platform differences, line numbers, and timestamps.

Unfortunately, they have a few limitations and pitfalls:

  • The value parser for these headers is very lax, and when it does fail to parse, the header is silently ignored.
    • #126370
    • Other headers have similar problems: #123760
  • The parser doesn't require a colon : after the header name, so the test suite contains an inconsistent mix of with-colon and without-colon rules.
    • #126777
  • The rules cannot contain literal double-quote " characters, which is very awkward for some kinds of output (e.g. JSON).
  • The parser uses parse_cfg_name_directive to help read the header name, which is sometimes useful (e.g. for defining separate rules for 32-bit and 64-bit), but most of the time it means that tests have to awkwardly write normalize-stderr-test instead of the more intuitive normalize-stderr.
  • The rules are applied to both compile output and run output, which is usually not a problem in practice, but can be unexpected.
  • Like many other headers, they are only meaningful in ui-mode tests, and are silently ignored if they appear in other kinds of test.

@rustbot label +A-compiletest

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue concerns compiletest's //@ normalize-* header parsing and lists several limitations, with some items already linked to #126370, #126777, and #123760. Start by locating the parser and the UI-mode tests that use these headers, then select an unchecked limitation, define its expected behavior, and add regression coverage showing that the chosen issue is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.