Add affected-test selection to `dotnet test`
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 1.3k
- PR merge metrics
- PR metrics pending
Description
### Is your feature request related to a problem?
Large test suites make local and pull-request validation expensive because `dotnet test` normally executes every selected test module and test. Teams can build test-to-source mappings and use those mappings to run only tests affected by a code change, but today this requires a separate launcher, separate project/module discovery, and a separate command-line experience.
An internal proof of concept has demonstrated this workflow:
1. Execute tests and record which source files and binaries are exercised by each test.
2. Persist that mapping as a reusable source of truth.
3. Compare the current Git changes with the stored mapping.
4. Discover the current tests and run only the affected subset.
The proof of concept currently exposes this through separate `collect` and `apply` commands. This duplicates responsibilities that `dotnet test` already owns, including project/solution build and evaluation, test-module resolution, process management, output, diagnostics, cancellation, and result reporting.
### Describe the solution you'd like
Add two explicit options to the Microsoft.Testing.Platform (MTP) implementation of `dotnet test`:
```console
dotnet test --refresh-test-mappings
dotnet test --run-affected-tests
```
`--refresh-test-mappings` would run the selected tests and create or update the persisted source-to-test mappings. The name intentionally describes maintaining the source of truth rather than exposing the proof of concept's implementation term, `collect`.
`--run-affected-tests` would compare repository changes with the latest compatible mappings, select the affected tests, and run them. "Affected tests" is intended to be understandable without prior knowledge of the industry term "Test Impact Analysis" and avoids colliding with the existing `--filter` concept.
#### Proposed behavior
- The feature is initially available only when `dotnet test` uses `Microsoft.Testing.Platform`.
- VSTest support is out of scope for the initial implementation and can be considered later based on demand and architectural fit.
- Existing `dotnet test` project/solution selection, build, target-framework, configuration, environment, results, diagnostics, output, and parallelism options remain authoritative.
- Project and solution inputs use the existing SDK build/evaluation path rather than rediscovering test assemblies through a second launcher.
- `--refresh-test-mappings` and `--run-affected-tests` are mutually exclusive.
- The repository must provide affected-test configuration and a compatible implementation package. The CLI should not implicitly download tooling.
- If mappings are missing or incompatible, `--run-affected-tests` should fail with actionable guidance rather than silently skipping tests. Whether a configurable "run all tests" safety fallback is desirable remains an open question.
- Conservative selection remains the default: changes outside the configured source scope should cause all tests for the relevant module to run rather than risk missing regressions.
- Normal `dotnet test` behavior remains unchanged when neither option is present.
#### Configuration and phased scope
The internal proof of concept provides a useful starting point for the feature scope.
The configuration needs to express:
- source files to include or exclude from impact tracking;
- files whose changes can be ignored, such as documentation;
- product modules to include or exclude from instrumentation;
- test modules to include or exclude;
- mapping storage and a context/tag that distinguishes configurations such as OS and architecture;
- execution tuning, diagnostics, result reporting, coverage, and native instrumentation where supported.
Proposed initial vertical slice:
- managed MTP test applications;
- project and solution inputs, including multi-targeted projects;
- SDK-resolved test modules and launch context;
- Git-based change detection;
- filesystem mapping storage;
- both mapping refresh and affected-test execution, so the scenario works end to end;
- normal MTP test filtering, environment variables, diagnostics, attachments, result aggregation, and cancellation;
- safe behavior for missing, stale, or incompatible mappings.
Potential follow-up scope:
- Azure Blob Storage, including local caching and standard Azure credential flows;
- Azure DevOps artifact storage;
- managed and native instrumentation;
- code coverage and TRX generation where these are not already better provided by MTP/`dotnet test`;
- running the discarded/non-affected tests as a second phase after affected tests pass;
- listing affected tests without executing them;
- build-log-based propagation from build inputs to generated binaries;
- retry and advanced batching policies;
- VSTest support.
#### High-level design
This should provide one user-facing launcher without loading the affected-test engine directly into `dotnet.dll`.
1. `dotnet test` continues to own command parsing, build/evaluation, project and solution semantics, module resolution, output, aggregate results, exit codes, and Ctrl+C behavior.
2. An optional, repository-pinned MTP affected-test worker package owns mapping collection, Git/storage analysis, affected-test selection, instrumentation, and storage providers.
3. The SDK starts that worker out of process. This isolates the SDK from the proof of concept's large managed/native dependency graph and lets the affected-test engine evolve and service independently.
4. The SDK sends the worker complete resolved module launch descriptors, not only assembly paths. These descriptors need the executable, arguments, working directory, project/TFM/RID or device identity, launch-profile contribution, environment variables, and architecture-specific `DOTNET_ROOT` behavior.
5. The worker owns MTP test-application processes for this mode and uses MTP JSON-RPC server mode for discovery and selected-test execution (`testing/discoverTests` and `testing/runTests`). This reuses MTP's existing bidirectional control protocol instead of extending the normal `dotnet test` reporting pipe into a second test-control protocol.
6. The SDK and worker communicate through a versioned streaming contract, preferably JSON-RPC over redirected standard I/O or a current-user-only local pipe. The contract carries initialization/capabilities, operation configuration, progress, test results, attachments, cancellation, partial failures, and completion.
7. Worker events are adapted into the existing `dotnet test` terminal reporter so affected-test mode preserves familiar progress, failures, summaries, artifacts, and exit behavior.
8. First Ctrl+C requests cooperative cancellation through the worker to MTP; a second Ctrl+C terminates the complete worker/test-host process tree.
The implementation package could expose its worker path and contract version through MSBuild properties/items. For a solution, all participating projects would need to resolve one compatible worker version. An environment-variable worker-path override could support development but should not be the production acquisition mechanism.
#### Compatibility and versioning
There are three contracts to version deliberately:
- SDK-to-worker control and event protocol;
- worker-to-MTP JSON-RPC compatibility;
- persisted test-mapping format.
The CLI should perform an initialization/capability handshake and report clear errors for unsupported worker or mapping versions. Mapping keys also need to account for target framework, runtime/device, OS, architecture, and other inputs that can change test behavior.
#### Suggested implementation stages
1. Define the CLI semantics, configuration discovery, package acquisition, version contracts, safety/fallback policy, and filter precedence.
2. Refactor the proof-of-concept command parsing away from its reusable mapping/selection engine and produce an MTP-only worker.
3. Prototype one SDK-resolved MTP project over a streaming worker contract, including reporting and process-tree cancellation.
4. Add the SDK execution strategy and fake-worker contract tests.
5. Deliver an end-to-end filesystem-storage implementation supporting both options.
6. Expand storage, coverage/native instrumentation, discarded-test execution, and other affected-test capabilities.
7. Consider VSTest support separately.
### Alternatives you've considered
- **Keep a standalone launcher:** preserves duplicated build/module discovery, output, cancellation, and command-line concepts. A standalone diagnostic tool may remain useful, but should not be the primary user experience.
- **Make `dotnet test` an alias for the proof-of-concept command:** useful for early validation, but it would hide rather than remove the duplicate launcher architecture and would not naturally preserve SDK project/solution semantics.
- **Load the affected-test engine directly into `dotnet.dll`:** not recommended because the proof of concept brings storage clients, Git/MSBuild/test-platform dependencies, code-coverage components, and native instrumentation payloads. This would increase SDK size and dependency/version risk and couple its servicing to the SDK release train.
- **Extend the current `dotnet test` named-pipe reporting protocol:** affected-test selection needs discover, select, and run-selected-tests control already modeled by MTP JSON-RPC. Adding parallel control semantics to the reporting pipe would create another protocol and require coordinated SDK/testfx evolution.
- **Implement this entirely as an MTP extension inside each test application:** insufficient for repository-wide Git/storage analysis and coordination across projects, modules, target frameworks, and processes.
- **Use `--test-selection` or `--test-impact`:** `--test-selection` can be confused with `--filter`, while "test impact" is established terminology but less self-explanatory. The proposed operation-specific options state exactly what they do.
### Additional context
An internal proof of concept has validated the mapping-refresh and affected-test workflow, conservative selection behavior, filesystem and remote storage, batching, managed/native instrumentation, coverage, and result reporting.
The initial proposal deliberately targets MTP because its JSON-RPC server mode already supports discovery followed by execution of selected test-node IDs. VSTest remains a potential future extension rather than a requirement for the first design.
A useful proof-of-concept milestone is: build and evaluate one MTP project through the existing SDK path, send its full launch descriptor to a minimal worker, discover tests through MTP JSON-RPC, execute an explicit subset, stream results into the existing terminal reporter, and verify that Ctrl+C terminates the complete process tree.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing `dotnet test` SDK path and Microsoft.Testing.Platform JSON-RPC server mode described in the proposal. Trace how resolved project launch descriptors, terminal reporting, and cancellation are handled before reviewing the suggested fake-worker contract tests. Done means an end-to-end MTP project can refresh mappings and run an affected subset through the existing reporter, with safe version and cancellation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, git
- Domain
- build-system, cli, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100