rust-lang / rust-lang/rust

compiletest output on message mismatch is hard to interpret

Open
#144,590 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-compiletest C-bug T-compiler
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

I find the new compiletest output on message mismatches quite hard to interpret. As an example:

---- [ui] tests/ui/pattern/usefulness/unions.rs stdout ----

error: 2 unexpected diagnostics reported, 2 expected diagnostics not reported
--- reported in JSON output but not expected in test file ---
tests/ui/pattern/usefulness/unions.rs:22:15: ERROR: non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
  expected with message non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
tests/ui/pattern/usefulness/unions.rs:28:15: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]
  expected with message non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered
---
--- expected in test file but not reported in JSON output ---
tests/ui/pattern/usefulness/unions.rs:22:13: ERROR: non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
  reported with message non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
  reported with kind NOTE with message patterns `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered
  reported with kind NOTE with message the matched value is of type `U8AsBool`
tests/ui/pattern/usefulness/unions.rs:28:13: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered
  reported with message non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]
  reported with kind NOTE with message patterns `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered
  reported with kind NOTE with message the matched value is of type `(U8AsBool, bool)`
---

Screenshot (with colors):

Image

Several things are confusing here:

  • This error has two parts, "reported in JSON output but not expected in test file" and "expected in test file but not reported in JSON output". However, the headers separating those parts are hard to spot since they are indented with these ---. That makes the tests/ui/pattern/usefulness/unions.rs:22:15 look like more of a header than the actual header. The colors actually make this worse since the black filename stands out much more than the green title.
  • For the "reported in JSON output but not expected in test file", there are then lines saying "expected with message". So the error wasn't expected but also it was expected? I think what this means to say is that another message was expected on the same line but I had to read the .rs file to understand that.
  • Similarly for the "expected in test file but not reported in JSON output", it then says "reported with kind" there. So something didn't get reported but also did get reported.
  • Showing all the messages reported on that line is too much and drowns the relevant output.

As an example of output I find much easier to interpret, here's an output mismatch printed by ui_test:

error: `constructing valid value` not found in diagnostics on line 3
 --> tests/fail/intrinsics/zero_fn_ptr.rs:3:62
  |
3 |     let _ = unsafe { std::mem::zeroed::<fn()>() }; //~ERROR: constructing valid value
  |                                                              ^^^^^^^^^^^^^^^^^^^^^^^^ expected because of this pattern
  |

error: there were 1 unmatched diagnostics
 --> tests/fail/intrinsics/zero_fn_ptr.rs:3:22
  |
3 |     let _ = unsafe { std::mem::zeroed::<fn()>() }; //~ERROR: constructing valid value
  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^ Error: Undefined Behavior: constructing invalid value: encountered a null function pointer
  |
Image

Maybe showing the full span is overdoing it, but the structure is at least much more clear.

Here's a possible incremental improvement of the current output that's already a lot easier to read for me:

error: 2 unexpected diagnostics reported
tests/ui/pattern/usefulness/unions.rs:22:15: ERROR: non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
  a different message was expected here: non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
tests/ui/pattern/usefulness/unions.rs:28:15: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]
  a different message was expected here: non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered

error: 2 expected diagnostics not reported
tests/ui/pattern/usefulness/unions.rs:22:13: ERROR: non-exhaustive patterns: `U8AsBool { n: 0_u8 }` and `U8AsBool { b: false }` not covered
  a different message was reported here: non-exhaustive patterns: `U8AsBool { n: 0 }` and `U8AsBool { b: false }` not covered [E0004]
tests/ui/pattern/usefulness/unions.rs:28:13: ERROR: non-exhaustive patterns: `(U8AsBool { n: 0_u8 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0_u8 }, false)` and 1 more not covered
  a different message was reported here: non-exhaustive patterns: `(U8AsBool { n: 0 }, false)`, `(U8AsBool { b: false }, false)`, `(U8AsBool { n: 0 }, false)` and 1 more not covered [E0004]

This got introduced in https://github.com/rust-lang/rust/pull/140622.
Cc @petrochenkov @jieyouxu

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining compiletest's diagnostic mismatch output using tests/ui/pattern/usefulness/unions.rs, then compare the current structure with the ui_test example in the issue. The work is done when mismatches clearly separate unexpected and missing diagnostics and identify differing messages without listing distracting extra messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.