rust-lang / rust-lang/rust

rustdoc doctest UI suite shouldn't need to manually normalize $DIR

Open
#149,616 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-doctests A-testsuite C-bug T-bootstrap T-rustdoc
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

Summary

I wrote the following UI test:

tests/rustdoc-ui/doctest/force-merge
//@ check-pass
//@ edition: 2018
//@ compile-flags: --test --test-args=--test-threads=1 --merge-doctests=yes -Z unstable-options
//@ normalize-stdout: "tests/rustdoc-ui" -> "$$DIR"
//@ normalize-stdout: "finished in \d+\.\d+s" -> "finished in $$TIME"
//@ normalize-stdout: "ran in \d+\.\d+s" -> "ran in $$TIME"
//@ normalize-stdout: "compilation took \d+\.\d+s" -> "compilation took $$TIME"
//@ normalize-stdout: ".rs:\d+:\d+" -> ".rs:$$LINE:$$COL"

// FIXME: compiletest doesn't support `// RAW` for doctests because the progress messages aren't
// emitted as JSON. Instead the .stderr file tests that this contains a
// "merged compilation took ..." message.

/// ```
/// let x = 12;
/// ```
///
/// These two doctests should be force-merged, even though this uses edition 2018.
///
/// ```
/// fn main() {
///     println!("owo");
/// }
/// ```
pub struct Foo;

In particular, note //@ normalize-stdout: "tests/rustdoc-ui" -> "$$DIR". That was cargo-culted from other tests in rustdoc.

This worked ok locally and generated a .stderr file with

test $DIR/doctest/force-merge.rs - Foo (line 14) ... ok
test $DIR/doctest/force-merge.rs - Foo (line 20) ... ok

It failed in CI because of normalization differences:

-	test $DIR/doctest/force-merge.rs - Foo (line 14) ... ok
-	test $DIR/doctest/force-merge.rs - Foo (line 20) ... ok
+	test $DIR/force-merge.rs - Foo (line 14) ... ok
+	test $DIR/force-merge.rs - Foo (line 20) ... ok

	Note: some mismatched output was normalized before being compared
-	test /checkout/tests/rustdoc-ui/doctest/force-merge.rs - Foo (line 14) ... ok
-	test /checkout/tests/rustdoc-ui/doctest/force-merge.rs - Foo (line 20) ... ok
+	test $DIR/force-merge.rs - Foo (line 14) ... ok
+	test $DIR/force-merge.rs - Foo (line 20) ... ok

That second note is really weird! Those are absolute paths. I don't see absolute paths when I run rustdoc locally, and I looked at the rustdoc invocation locally and CI and didn't notice any interesting differences. The only thing I can think of is something related to remap-path-prefix, but that job didn't set remap-path-prefix.

I think what happened is that the absolute path got normalized by compiletest, not by the test directive: https://github.com/rust-lang/rust/blob/ab196d7a7de1170ac2b376fb343f633d29d75685/src/tools/compiletest/src/runtest.rs#L2557-L2558

Command used
./x t --stage 1 tests/rustdoc-ui/doctest/force-merge.rs
Expected behaviour

Compiletest normalizes the path to the UI directory automatically.

Actual behaviour

Compiletest normalizes the path in CI, but not locally.

Bootstrap configuration (bootstrap.toml)
profile = "compiler"
change-id = 102579

[rust]
lld = true
deny-warnings = false

download-rustc = true
debug-assertions = false
verbose-tests = true

[build]
description = "\u001b[0;95mlove you love you love you\u001b[0m"
Operating system

macOS Sequoia 15.5 (24F74)

HEAD

ab196d7a7de1170ac2b376fb343f633d29d75685

Additional context

Extracted from https://github.com/rust-lang/rust/pull/149565#issuecomment-3608564732.

@rustbot label T-rustdoc A-testsuite A-doctests

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run ./x t --stage 1 tests/rustdoc-ui/doctest/force-merge.rs and inspect src/tools/compiletest/src/runtest.rs around lines 2557-2558. Compare local and CI normalization of the paths shown in the issue, then verify the test's .stderr output no longer needs the manual tests/rustdoc-ui-to-$DIR directive and is consistent in both environments.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.