[RFC] Agent/LLM-efficient test output for Microsoft.Testing.Platform

## Summary

Open an umbrella discussion to track LLM- and agent-efficiency improvements for **Microsoft.Testing.Platform (MTP)** and **MSTest** output.

LLMs and coding agents (Copilot, Claude Code, Cursor, Codex, Aider, …) are increasingly the primary readers of `dotnet test` output. Each token of noise we emit translates into context-window pressure, slower agent loops, and more LLM cost. The repo already has scaffolding for this (`LLMEnvironmentDetector`, LLM-aware `--show-stdout`/`--show-stderr` defaults via #8771, `--list-tests json` via #8280, `--no-ansi`/`--no-progress` flags via #7647/#7649), but several large wins are still missing.

This RFC-style issue collects the candidate improvements and links to focused sub-issues so they can be discussed, prioritized, and shipped individually.

## Motivation

A typical agent flow today looks like:

1. Agent runs `dotnet test` (or `dotnet run --project ...` for MTP).
2. The test host prints a banner, runtime info, a live progress region, and per-test outcome lines (~all in color).
3. On failure, each failing test prints message + expected/actual + stack trace including ~10 MTP/MSTest plumbing frames.
4. The agent must reread the whole transcript to figure out which tests failed and where.

Symptoms an agent sees today, all of which cost tokens:

- ANSI color escape codes are emitted in non-LLM CI runs unless `NO_COLOR`-equivalent flags are set. The de-facto industry standard env var [`NO_COLOR`](https://no-color.org) is **not** honored by name.
- Stack traces include the full MTP plumbing chain (`Microsoft.Testing.Platform.Hosts.*`, `Microsoft.Testing.Platform.MSTest.*`, `Microsoft.VisualStudio.TestPlatform.MSTestAdapter.*`, …). User code is only a handful of frames; the rest is noise.
- There is no compact "what failed and where" footer, so the agent has to re-parse the whole log.
- There is no first-class machine-readable test-results output (TRX is XML and verbose; `--list-tests json` only covers discovery; HTML report is not parseable). Each failure block is paragraph-style text.
- `--show-stdout` / `--show-stderr` help text still claims default is `All` even though it flips to `Failed` in LLM mode (tracked at #8772, still open).
- `dotnet test`/MSBuild wrapping can hide MTP's failure detail behind `TestingPlatformShowTestsFailure=false` (see #5188).

## Detailed proposals

Each item below should be filed as a separate issue (linked when filed) so it can be designed, sized, and shipped independently.

### Output: noise reduction

- [x] **Honor `NO_COLOR` env var natively** (industry standard, see ). Treat any non-empty `NO_COLOR` like `--no-ansi`. `--ansi on|off|auto` should still take precedence.
→ #8825

- [ ] **Filter MTP/MSTest internal stack frames** in the terminal failure output. Hide frames whose declaring type lives under `Microsoft.Testing.Platform.*` (non-public namespaces) or `Microsoft.VisualStudio.TestPlatform.MSTestAdapter.*`. Keep user-code frames. Apply at the rendering layer (`FormatStackTrace`/`AppendStackFrame` in `TerminalTestReporter.Formatting.cs`) so raw exception data flowing to TRX/HTML/AzDO reports is untouched. Default-on in LLM mode, opt-in flag (`--full-stack-trace`) for humans who want everything.
→ #8826 — **closed as not planned** (decided against: internal MTP/MSTest frames are kept for debuggability)

- [ ] **Deduplicate identical failures across tests.** Group failures whose `(exception type + first N stack frames + first line of message)` are identical; print details once, reference subsequent failures by count + list of test ids. Reduces N×output bloat when a shared helper breaks.
→ sub-issue: _to be filed_

- [x] **Crash/hang dump output volume** — verify #7045 fix scope; sometimes a dump still prints multi-screen native stacks before the test summary.

### Output: machine-readable / structured

- [ ] **`Microsoft.Testing.Extensions.JsonReport` NDJSON event stream** (new extension, modeled on `TrxReport`/`HtmlReport`/`AzureDevOpsReport`). One JSON object per test event (started/completed) + a final summary. Stable schema documented in `PACKAGE.md`. Inspired by Jest's `--json`, pytest-json-report, and Vitest's JSON reporter.
→ #8828

- [ ] **Compact end-of-run failure index footer** (sentinel-delimited block listing only `Test FQN | first-line error | file:line` per failure). Agents can grep that block alone instead of re-reading the full log. Default-on in LLM mode only, no CLI flag to avoid help/info-snapshot churn for the first revision.
→ #8827

- [ ] **OpenTelemetry sanitization** — sanitize raw paths/PII/stack traces before export (#8411 already open).

### Input: agent ergonomics

- [x] **Fix `--show-stdout`/`--show-stderr` help text** to honestly describe the LLM-mode-aware default (#8772 already open). Needs XLF regen + help/info acceptance updates.

- [x] **Reduce default banner/preamble in LLM mode.** "Microsoft (R) Testing Platform vX.Y.Z" + runtime info + per-extension banners cost tokens before any test even runs. In LLM mode, skip or compress to a single line.

- [ ] **`--help --output json` / `--info --output json`** for machine-readable help/info. Agents can query for the option they need without scanning ~160 lines of text.

- [ ] **Stable error codes for common misconfigurations.** Today, errors like "option X requires Y" are English-only — give them codes (e.g. `MTP1001`) so agents can match without prompt-engineering.

### `dotnet test` / MSBuild wrapping

- [ ] **Agent-friendly `dotnet test` defaults / documentation.** `TestingPlatformShowTestsFailure=false` by default in `Microsoft.Testing.Platform.MSBuild.targets` means agents only see a pointer to a log file unless they opt in. Flip default to `true` when an LLM env is detected (or document the env var to flip it). Cross-link #5188.

### Documentation

- [ ] **Public doc page: "Microsoft.Testing.Platform for AI agents."** Document `LLMEnvironmentDetector` behavior, detected env vars, `NO_COLOR`, recommended flags, JSON report extension.

- [ ] **`AGENTS.md` at repo root** referencing `.github/copilot-instructions.md` (industry convention emerging in 2025).

## Drawbacks

- Each item changes some part of the terminal transcript and risks breaking log-scraping CI consumers. Mitigations: gate behavioral changes to LLM-mode-only by default, expose opt-in/opt-out flags for humans, and keep raw report data (TRX/HTML/AzDO) untouched.
- Adding a new JSON report extension grows the package surface and the maintenance footprint.
- Stack-trace filtering can hide a frame that's actually relevant to extension authors debugging MTP itself; needs an opt-out path.

## Alternatives

- Do nothing and rely on agents to learn to ignore the noise. Trades user tokens/cost.
- Wait for the .NET SDK to standardize an agent-friendly test output. Doesn't help today.

## Compatibility

Not a breaking change if behavioral defaults stay LLM-mode-gated and human-mode output is unchanged.

## Unresolved questions

- Should the JSON report extension be packaged separately or rolled into an existing one?
- Should NO_COLOR support also imply `--no-progress`, or stay color-only as the spec says?
- Should the failure-index footer be a flag (`--failure-index` / `--no-failure-index`) or env var, or strictly LLM-mode-gated for v1?

## References

- `src/Platform/Microsoft.Testing.Platform/Helpers/LLMEnvironmentDetector.cs`
- `src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.cs` (LLM-aware behavior switches)
- `src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/ExceptionFlattener.cs` (stack trace passthrough)
- `src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.Formatting.cs` (stack-frame rendering)
- Related: #7647, #7649, #8084, #8280, #8411, #8771, #8772, #5188, #7045
- External: , [Jest `--json`](https://jestjs.io/docs/cli#--json), [pytest-json-report](https://pypi.org/project/pytest-json-report/), [Vitest reporters](https://vitest.dev/guide/cli.html#reporters)

> Filed by @Evangelink based on an analysis of the MTP/MSTest output surfaces from an agent-consumer perspective.

This is an umbrella RFC rather than a single change. Start with Helpers/LLMEnvironmentDetector.cs and OutputDevice/TerminalOutputDevice.cs, then read the reporter files named in the proposal and the linked focused issues. Done means selecting one proposal with a scoped issue, defined behavior, and acceptance tests; the RFC itself does not provide one implementation path.

Written by the indexing model from the issue text.