rust-lang / rust-lang/rust

Failures in combined doctests with a test runner are difficult to diagnose

Open
#141,064 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-doctests A-edition-2024 C-bug I-edition-triaged T-rustdoc
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

When using 2024 combined doctests and a test runner, and a test causes a process failure (like a crash), it can be difficult to determine which test caused the problem.

Example:

/// ```
/// let x = 1;
/// ```
pub fn a() {}

/// ```
/// std::process::abort();
/// ```
pub fn b() {}

/// ```
/// let x = 1;
/// ```
pub fn c() {}

With the following shell script called runner.sh:

#!/bin/bash

echo "$@"

eval "$@"

Running the command:

rustdoc --edition=2024 --test --test-runtool=./runner.sh foo.rs

results in the following output:

/var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestVI7Kq5/rust_out
WARNING: No rustdoc doctest environment variable provided so doctests will be run in the same process

running 3 tests
/Users/eric/Temp/z36/./runner.sh: line 5: 72864 Abort trap: 6           /var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestVI7Kq5/rust_out

There's no way to tell from that output which of the tests caused the problem.

If you run with 2021 edition (or 2024 without a test runner), you get a more reasonable output:

running 3 tests
test foo.rs - c (line 11) ... ok
test foo.rs - a (line 1) ... ok
test foo.rs - b (line 6) ... FAILED

failures:

---- foo.rs - b (line 6) stdout ----
Test executable failed (exit status: 134).

stdout:
/var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestj1EgFx/rust_out

stderr:
/Users/eric/Temp/z36/./runner.sh: line 5: 73368 Abort trap: 6           /var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestj1EgFx/rust_out



failures:
    foo.rs - b (line 6)

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s

It would be nice if there would be some way to catch this.

Meta

rustc --version --verbose:

rustc 1.89.0-nightly (414482f6a 2025-05-13)
binary: rustc
commit-hash: 414482f6a0d4e7290f614300581a0b55442552a3
commit-date: 2025-05-13
host: aarch64-apple-darwin
release: 1.89.0-nightly
LLVM version: 20.1.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The entry points in the report are rustdoc --edition=2024 --test --test-runtool=./runner.sh and foo.rs, with runner.sh reproducing the process failure. Reproduce that command first, then trace combined doctest reporting; done means the output identifies which doctest failed when a test-runner process aborts.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, rust
Domain
compilers, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.