Implement Opt-In Documentation Tests on Specified Code Blocks
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 1.3k
- PR merge metrics
- PR metrics pending
Description
### Is your feature request related to a problem? Please describe.
Documentation examples in XML doc comments frequently drift from reality as APIs evolve. Even in well-maintained libraries, it’s easy for `` examples in `///` comments to become outdated because they are not compiled nor executed as part of `dotnet build` / `dotnet test`.
This causes:
* Broken examples shipped to users in IntelliSense / generated docs.
* Extra maintenance cost (manual validation, extra development time, bugs filed by users).
* Reduced confidence in docs for SDK style and library authors.
Rust’s doctest model (doc examples are compiled/run as tests) keeps documentation examples correct. An equivalent, first-class or standardized tool does not exist in the .NET SDK toolchain that compiles/runs doc comment code examples as part of the normal pipeline.
---
### Describe the solution you'd like
* Add an **opt-in** capability in the .NET SDK toolchain to treat marked documentation `` snippets as “doctests” that can be compiled (and optionally executed) via standard CLI workflows.
* Goal: dotnet test should validate doctest snippets when enabled.
##### **MVP / minimal solution (compile-only):**
* When enabled, extract code snippets marked as doctest (e.g. `...`) from the project’s XML documentation output and compile them during `dotnet test` (or as a dedicated target).
* Default behavior is **off**.
* Doctest failures should show up as test failures with useful diagnostics (API Member + snippet index / snippet location and compilation errors).
##### **Authoring / marking snippets:**
* Use a tool-specific marker compatible with existing XML doc comments. e.g.
* `/// ... `
##### **Configuration (SDK-native):**
* Enable per project via MSBuild properties (e.g. in `.csproj`), and allow repo-wide defaults via `Directory.Build.props`.
* Proposed Options:
* `DoctestEnabled` = `true` | `false` [default: `false`]
* `DoctestMode` = `Compile` | `Run` [default: `Compile`]
* `DoctestInclude` / `DoctestExclud`e = semicolon-separated patterns matching member IDs (and optionally snippet tags) [defaults to include al]l (Exclude is prioritized over Include)
* `DoctestTimeoutSeconds` = `X` [default: 10] (if/when `Run` is enabled)
* Safety controls [default will be on, opt-out] (e.g. timeouts, isolation, disabling network/IO by default).
##### **Execution options (more than happy and open for guidance):**
A) Generate temporary wrapper sources in `obj/` and execute via the existing test infrastructure
B) Integrate as a discovery/execution layer for `dotnet test` (VSTest and/or Microsoft Testing Platform) so doctests appear as native tests without requiring a specific test framework.
##### **Other solutions considered:**
* External tools (custom scripts, third-party doctest extractors) work but aren’t standardized, require bespoke setup, have varying maintenance/quality, and/or don’t feel integrated with the SDK/test pipeline.
* Roslyn analyzers can help with editor diagnostics but don’t naturally provide “run with `dotnet test`” without additional runner integration.
---
### Additional context
##### Why in dotnet/sdk?
* The desired UX is “opt-in, but when enabled it feels like a normal SDK capability”: controlled via MSBuild properties / Directory.Build.props, and exercised through standard CLI commands (`dotnet test`, perhaps with a tag, and optionally `dotnet build`). The .NET SDK is responsible for the default build pipeline integration and is the natural place to wire an officially supported workflow without requiring a new language feature.
##### Goals:
* Keep documentation examples correct by compiling them in CI.
* Make the “happy path” easy for library authors: `dotnet test` validates doc examples when enabled.
* Ensure compatibility with existing doc comment ecosystems: markers should be plain XML that doesn’t break doc generation or IntelliSense.
#### Non-goals:
* No new C# language syntax required.
* No requirement that all `` blocks become doctests; only explicitly marked ones that are not filtered out.
* Running doctests (`Run` mode) can be a later/optional phase; compile-only is valuable on its own and is my priority.
##### Open questions / guidance requested:
* Best integration point within the SDK: MSBuild target vs tighter `dotnet test` runner integration.
* Recommended default behavior, configuration, and preferred implementation details if any.
* This is my first time filing a feature request here - I'm more than happy to adjust scope/approach to match team preferences.
---
### Potential Examples:
* Use:
* `doctestTags` is optional and can be omitted from MVP.
```csharp
/// Parses a widget.
///
///
/// var w = Widget.Parse("abc");
/// Console.WriteLine(w.Value);
///
///
public static Widget Parse(string s) => ...
```
* Enabling via `.csproj`
```xml
true
true
Compile
```
* Enabling repo-wide defaults:
```xml
false
Compile
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how .csproj and Directory.Build.props settings flow into XML documentation generation and the dotnet test pipeline. Compare the proposed MSBuild-target and test-runner integration points, then define completion around opt-in marked snippets, compile-only validation, useful failures, and the stated configuration defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- build-system, cli, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100