rustdoc: Tracking issue for injecting doctests into the host crate

[!NOTE]

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.

A tracking issue is however not meant for large scale discussion, questions, or bug reports.
Instead, open a dedicated issue for the specific matter or post in the #t-rustdoc Zulip channel.

Discussion comments will get marked as off-topic or deleted.
Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

From: https://hackmd.io/@fmease/inject-doctests-into-host-crate.
The original idea was proposed by @Urgau.

This is a greenlit experiment that might not pan out!

Motivation

• Have doctest work in binary crate
• Have better warnings and errors reporting by having correct spans
• Handle doctests with different editions in the same compilation unit
• Simplify cross-compilation (deprecating all the --doctest-cross-arg ...)
• Better Cargo integration (parallel test exec, smarter unused_crate_dependencies, ...)

Implementation

• Have a hook in the AST->HIR lowering that is only implemented by rustdoc:
  • It would parse the doctests, create the AST nodes (by calling rustc_parse maybe?) and give back to rustc the AST nodes corresponding to #[test] fn doctest_1234 { ... } with the doctest body inside
• Rustdoc would give spans corresponding to the original doctest code in the doc comment. This could include a different edition for doctests that override that
  • Problem: how would we get the spans to be correct given the presence of /// in the original code but not the new code given to rustc_parse? would it just work or does it need more invasive changes?
• These AST nodes would then be injected into the crate and lowered into HIR and go through the rest of the compilation
• Then we run them using the libtest harness.
• We need to run all rustc passes to make it work

Implementation History

Empty.

Start with the linked HackMD design and the rustdoc AST-to-HIR lowering hook described in the issue; no files or tests are named. Define the experiment's design around parsing doctests into test AST nodes, preserving spans and editions, running rustc passes, and executing them with the libtest harness.