Failures in combined doctests with a test runner are difficult to diagnose
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 119k
- Forks
- 16.1k
- PR merge metrics
- PR metrics pending
Description
When using 2024 combined doctests and a test runner, and a test causes a process failure (like a crash), it can be difficult to determine which test caused the problem.
Example:
/// ```
/// let x = 1;
/// ```
pub fn a() {}
/// ```
/// std::process::abort();
/// ```
pub fn b() {}
/// ```
/// let x = 1;
/// ```
pub fn c() {}
With the following shell script called runner.sh:
#!/bin/bash
echo "$@"
eval "$@"
Running the command:
rustdoc --edition=2024 --test --test-runtool=./runner.sh foo.rs
results in the following output:
/var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestVI7Kq5/rust_out
WARNING: No rustdoc doctest environment variable provided so doctests will be run in the same process
running 3 tests
/Users/eric/Temp/z36/./runner.sh: line 5: 72864 Abort trap: 6 /var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestVI7Kq5/rust_out
There's no way to tell from that output which of the tests caused the problem.
If you run with 2021 edition (or 2024 without a test runner), you get a more reasonable output:
running 3 tests
test foo.rs - c (line 11) ... ok
test foo.rs - a (line 1) ... ok
test foo.rs - b (line 6) ... FAILED
failures:
---- foo.rs - b (line 6) stdout ----
Test executable failed (exit status: 134).
stdout:
/var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestj1EgFx/rust_out
stderr:
/Users/eric/Temp/z36/./runner.sh: line 5: 73368 Abort trap: 6 /var/folders/6l/84bdkbvx07zg94gyt3ryvlc40000gn/T/rustdoctestj1EgFx/rust_out
failures:
foo.rs - b (line 6)
test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s
It would be nice if there would be some way to catch this.
Meta
rustc --version --verbose:
rustc 1.89.0-nightly (414482f6a 2025-05-13)
binary: rustc
commit-hash: 414482f6a0d4e7290f614300581a0b55442552a3
commit-date: 2025-05-13
host: aarch64-apple-darwin
release: 1.89.0-nightly
LLVM version: 20.1.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The entry points in the report are rustdoc --edition=2024 --test --test-runtool=./runner.sh and foo.rs, with runner.sh reproducing the process failure. Reproduce that command first, then trace combined doctest reporting; done means the output identifies which doctest failed when a test-runner process aborts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, rust
- Domain
- compilers, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100