rust-lang / rust-lang/rust

`src/tools/rustdoc-gui-test` uses compiletest directives + `TestProp` handling internals in a questionable way

Open
#143,827 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-test-infra A-testsuite C-bug T-bootstrap T-rustdoc
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

I was wondering why compiletest's Config had a default, it turns out it's used by

https://github.com/rust-lang/rust/blob/9535feebd5741a55fc24e84060e82d41a75dac6e/src/tools/rustdoc-gui-test/src/main.rs#L114-L137

to piggy-back off of compiletest's directive parsing and compile/run-flags extraction.

This happens to work because only //@ {compile,run}-flags directives are used, which do not otherwise happen to rely on other parts of the compiletest config or get validated in a way that requires other parts of the compiletest config (for now).

tests/rustdoc-gui/src/extend_css/lib.rs
1://@ compile-flags: --extend-css extra.css

tests/rustdoc-gui/src/theme_css/lib.rs
1://@ compile-flags: --theme custom-theme.css

tests/rustdoc-gui/src/scrape_examples/src/lib.rs
1://@ run-flags:-Zrustdoc-scrape-examples

tests/rustdoc-gui/src/link_to_definition/lib.rs
1://@ compile-flags: -Zunstable-options --generate-link-to-definition

IMO this is very questionable, because this bypasses any validation that compiletest might perform between parsing CLI args passed from bootstrap, and then constructing a validated Config. Furthermore, you'll appear to be able to use //@ only-xxx and other conditional test execution directives, where in reality they do not work at all (or are completely wrong) because the dummy config prevents compiletest conditional test directives from having any info to work with properly.

In other words, compiletest tries to be internally self-consistent / coherent, but it can't enforce that if its internals get exposed and "intercepted".

Possible mitigations / better approaches
  • If //@ {run,compile}-flags is all rustdoc-gui-test needs, it might be better to just reimplement a naive per-line //@ {compile,run}-flags.
  • Or try to merge rustdoc-gui-test into compiletest, but it runs tests with node in quite a different way versus other test suites.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/tools/rustdoc-gui-test/src/main.rs around lines 114-137, then inspect compiletest Config construction and directive parsing. Review the listed rustdoc-gui test files and determine whether the tool should parse only compile/run flags or be integrated with compiletest; done means avoiding an invalid dummy configuration and making unsupported directives unambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, rust
Domain
testing-qa, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.