Fallout-build / Fallout-build/Fallout
First-class Microsoft.Testing.Platform (MTP) support for `dotnet test`
- Dominant language
- C#
- Stars
- 154
- Forks
- 19
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 15
Description
## Summary
.NET 10 dropped VSTest support for MTP-native test projects (xunit.v3, MSTest/NUnit-with-MTP). Migrating a repo to `global.json`'s `"test": {"runner": "Microsoft.Testing.Platform"}` currently hits two distinct bugs in Fallout, plus a missing feature, all discovered while migrating a real repo (Azure Functions service + its own build-automation library, both xunit v2 → xunit.v3).
## 1. `FalloutBuild.GetBuildAssemblyFile()` throws under an MTP test host
`src/Fallout.Build/FalloutBuild.Statics.cs`:
```csharp
private static AbsolutePath GetBuildAssemblyFile()
{
Assembly entryAssembly = Assembly.GetEntryAssembly();
if (entryAssembly == null || entryAssembly.GetTypes().All((Type x) => !x.IsSubclassOf(typeof(FalloutBuild))))
{
string text = entryAssembly?.GetName().Name;
Assert.True(text == null || text.StartsWith("ReSharperTestRunner") || text == "testhost",
"Assembly name was " + StringExtensions.SingleQuote(text), ...);
return null;
}
...
}
```
This runs in `FalloutBuild`'s static constructor, so it fires the moment *anything* touches a static member of `FalloutBuild` (including `FalloutBuild.RootDirectory`, which several generated settings classes — e.g. `FunctionAppSettings`-style classes with `AbsolutePath` defaults — read in a field initializer). Under classic VSTest, the entry assembly is `testhost`/`testhost.exe`, which the allowlist recognizes. Under MTP, the entry assembly is the **test assembly itself** (MTP self-hosts it as the process entry point), so `entryAssembly.GetName().Name` is e.g. `MyLibrary.Specs` — matches none of the three hardcoded patterns — and the assert throws `TypeInitializationException` from inside completely unrelated unit tests that never reference `FalloutBuild` directly, only a settings class from a build-automation library that happens to depend on it.
**Suggested fix:** drop the assert entirely. If no type in the entry assembly derives from `FalloutBuild`, that's sufficient signal on its own — there's no need to also validate the entry assembly's *name* against a closed list that can never enumerate every current and future test host (VSTest's `testhost`, ReSharper's runner, MTP's self-hosted test assemblies, whatever comes next). Just `return null`.
## 2. No MTP-mode `dotnet test` support — `DotNetTestSettings` only knows the classic VSTest CLI surface
`src/Fallout.Common/Tools/DotNet/DotNet.json`'s `Test` definition generates `DotNetTestSettings` 1:1 from the classic (pre-.NET-10) `dotnet test` surface: `ProjectFile` is `"position": 1` (bare positional argument, no `--project` alternative), plus `Loggers` (`--logger`), `RunSettings` (`-- {key}={value}`), `BlameMode`/`BlameCrash*`, `DataCollector`, `TestAdapterPath` — all VSTest-only concepts.
Per Microsoft's own reference (https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test-mtp), MTP mode's `dotnet test` is a **different, non-overlapping CLI surface**:
- No positional project path at all. `--project`, `--solution`, and `--test-modules` are mutually exclusive and are the *only* documented ways to select what to test.
- `--logger`/`RunSettings`/`--blame*`/`--collect`/`--test-adapter-path` don't exist; coverage and TRX reporting are done via extension-registered flags instead (`--coverage`, `--coverage-output-format`, `--coverage-output` from `Microsoft.Testing.Extensions.CodeCoverage`; `--report-trx`, `--report-trx-filename` from `Microsoft.Testing.Extensions.TrxReport`).
- Extra MTP-only platform options with no VSTest equivalent: `--minimum-expected-tests`, `--diagnostic-output-directory`, `--treenode-filter`, `--zero-tests-policy`, etc.
Concretely, today, calling `.SetProjectFile(path)` on `DotNetTestSettings` under a repo pinned to the MTP runner produces a command dotnet's own CLI parser can reject outright: `Specifying a project for 'dotnet test' should be via '--project'.` (observed on SDK 10.0.105; later 10.0.4xx patches tolerate the positional form too, but it isn't part of MTP's documented contract, so relying on it is fragile across SDK versions).
**Suggested fix:** add a dedicated MTP-mode test task/settings class (generated from a new `TestMtp.json`, or an `"mtp": true` variant on the existing definition — whichever fits the generator better) with:
- `ProjectFile` / `SolutionFile` / `TestModules` → `--project {value}` / `--solution {value}` / `--test-modules {value}` (no positional option).
- `Coverage` (`--coverage`), `CoverageOutputFormat` (`--coverage-output-format {value}`), `CoverageOutput` (`--coverage-output {value}`).
- `ReportTrx` (`--report-trx`), `ReportTrxFileName` (`--report-trx-filename {value}`).
- `MinimumExpectedTests`, `DiagnosticOutputDirectory`, `TreenodeFilter`, `ZeroTestsPolicy` and other MTP-only platform options.
- Keep the properties that exist on both surfaces: `Configuration`, `Framework`, `Filter`, `Verbosity`, `NoBuild`, `NoRestore`, `ResultsDirectory`.
### Concrete `TestMtp.json`, same shape as the real `DotNet.json` `Test` entry
I decompiled `Fallout.Common.Tooling.ToolOptions.GetArgument` (`Fallout.Tooling.dll`) to check exactly how `Format`/`Position`/`Separator` combine for list-typed properties before proposing this, rather than guessing:
```json
{
"help": "Runs tests using the Microsoft.Testing.Platform (MTP) native runner, opted into via global.json's 'test.runner'. VSTest and MTP are separate, non-overlapping dotnet test CLI surfaces (see https://learn.microsoft.com/dotnet/core/tools/dotnet-test-mtp), so this is a distinct task from Test rather than an extension of it.",
"postfix": "TestMtp",
"commonPropertySets": ["restore", "restore-runtime"],
"definiteArgument": "test",
"settingsClass": {
"properties": [
{ "name": "ProjectFile", "type": "string", "format": "--project {value}", "help": "Path to the test project. Mutually exclusive with SolutionFile and TestModules." },
{ "name": "SolutionFile", "type": "string", "format": "--solution {value}", "help": "Path to the test solution. Mutually exclusive with ProjectFile and TestModules." },
{ "name": "TestModules", "type": "string", "format": "--test-modules {value}", "help": "Glob selecting already-built test module assemblies. Mutually exclusive with ProjectFile and SolutionFile." },
{ "name": "Configuration", "type": "string", "format": "--configuration {value}" },
{ "name": "Framework", "type": "string", "format": "--framework {value}" },
{ "name": "Filter", "type": "string", "format": "--filter {value}" },
{ "name": "Verbosity", "type": "DotNetVerbosity", "format": "--verbosity {value}" },
{ "name": "NoBuild", "type": "bool", "format": "--no-build" },
{ "name": "NoRestore", "type": "bool", "format": "--no-restore" },
{ "name": "ResultsDirectory", "type": "string", "format": "--results-directory {value}" },
{ "name": "MinimumExpectedTests", "type": "int?", "format": "--minimum-expected-tests {value}" },
{ "name": "DiagnosticOutputDirectory", "type": "string", "format": "--diagnostic-output-directory {value}" },
{ "name": "TreenodeFilter", "type": "string", "format": "--treenode-filter {value}" },
{ "name": "ZeroTestsPolicy", "type": "ZeroTestsPolicy", "format": "--zero-tests-policy {value}" },
{ "name": "Coverage", "type": "bool", "format": "--coverage", "help": "Requires the Microsoft.Testing.Extensions.CodeCoverage package." },
{ "name": "CoverageOutputFormat", "type": "string", "format": "--coverage-output-format {value}" },
{ "name": "CoverageOutput", "type": "string", "format": "--coverage-output {value}" },
{ "name": "ReportTrx", "type": "bool", "format": "--report-trx", "help": "Requires the Microsoft.Testing.Extensions.TrxReport package." },
{ "name": "ReportTrxFileName", "type": "string", "format": "--report-trx-filename {value}" },
{ "name": "ExtensionArguments", "type": "List", "format": "{value}", "prefix": "--", "position": -1, "help": "Raw arguments forwarded verbatim to registered MTP extensions, placed after a literal '--' as Microsoft's dotnet test docs recommend, to avoid argument-binding ambiguity with recognized options." }
]
}
}
```
`"prefix": "--"` is new — it doesn't exist in the schema today, and addresses issue 3 below. Two existing multi-value mechanisms come close but aren't quite right for raw-token forwarding: a `List` with `Separator` set joins all values into a single glued string token (wrong — a forwarded token can itself contain spaces, e.g. a file path, and needs to stay its own argv element); a `List` with `Separator == null` (like `Loggers`, format `"--logger {value}"`) repeats the *entire* format before every value, so a literal-only part (no `{value}`) gets emitted once per item, not once total. `prefix` is a minimal, generic, opt-in addition to `ArgumentAttribute`/the `GetListArguments()` branch of `ToolOptions.GetArgument`: when set and the list is non-empty, emit that one literal token before the per-value tokens (reusing the existing `Separator == null` per-value loop, just unshifted by one literal item first) — keeping every forwarded token its own argv entry.
### What this buys consumers
```csharp
DotNetTestMtp(s => s
.SetProjectFile(Solution.MyProject_Specs)
.SetConfiguration(Configuration.Debug)
.SetFilter("Category!=Integration")
.EnableReportTrx()
.SetReportTrxFileName(ArtifactsDirectory / "UnitTestResults.trx")
.EnableCoverage()
.SetCoverageOutputFormat("cobertura")
.SetCoverageOutput(ArtifactsDirectory / "UnitTests.cobertura.xml"));
```
— a typed, discoverable, IntelliSense-able API, matching the exact `Set`/`Enable`/`Disable` fluent conventions the generator already produces for every other tool, instead of the raw-string `AddProcessAdditionalArguments("--project", ..., "--coverage", ...)` workaround this issue's repro required.
## 3. No structured way to forward extension arguments safely
Microsoft's docs explicitly recommend putting extension-forwarded arguments after a literal `--`, because "When a recognized option appears between an unrecognized option name and its value, removing the recognized option can change how the leftover tokens bind." `RunSettings` already has the right shape for this (`Position = -1`) but is VSTest-runsettings-specific (`key=value` pairs), not a generic forwarder; the generic `ProcessAdditionalArguments` escape hatch on `ToolOptions` doesn't guarantee placement after `--` either. `ProcessAdditionalArguments` is already unconditionally concatenated at the very end of every generated argument list (confirmed in the decompiled `GetArguments()`), which is the right place — it just needs the `--` guarantee.
**Suggested fix:** the `ExtensionArguments`/`prefix` addition shown above, on the new MTP settings class specifically (rather than changing `ProcessAdditionalArguments` globally, since a literal `--` doesn't mean the same thing, or anything, for most other wrapped CLI tools).
## 4. (Secondary, not MTP-specific) `PublishCodeCoverage` silently accepts an unsupported glob
`AzurePipelines.Instance.PublishCodeCoverage(tool, summaryFile, reportDirectory)` takes `summaryFile` as a plain path with nothing in the signature indicating it can't be a glob — but the underlying `##vso[codecoverage.publish]` Azure Pipelines logging command has never supported wildcards. Passing e.g. `Artifacts/*.cobertura.xml` builds without complaint and fails only when the pipeline actually runs, with a cryptic agent-side "file does not exist" error. A guard clause (reject/throw on `*`/`?` in the path) or at least an XML-doc callout would turn this into a fail-fast, discoverable mistake instead of a pipeline-time surprise.
## Repro context
Hit all of the above migrating a private repo from xunit 2 + VSTest to xunit.v3 + MTP across three test projects (a service's unit tests, its own build-automation library's unit tests, and its end-to-end tests), on Fallout.Common/Fallout.Build 10.4.0.
Contributor guide
Assessment
This issue has not been assessed yet.