Add JSON / NDJSON test report extension (Microsoft.Testing.Extensions.JsonReport)
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 1k
- Forks
- 312
- Avg merge
- 8h 30m
- Merged PRs (30d)
- 469
Description
Summary
Introduce a new MTP extension Microsoft.Testing.Extensions.JsonReport that emits a stable, documented NDJSON (newline-delimited JSON) event stream of test execution events, plus a final summary event. Models the existing Microsoft.Testing.Extensions.TrxReport, Microsoft.Testing.Extensions.HtmlReport, and Microsoft.Testing.Extensions.AzureDevOpsReport extensions.
Motivation
Today MTP offers two machine-readable outputs:
--list-tests json(test discovery only, added in #8280)- TRX XML via
Microsoft.Testing.Extensions.TrxReport(verbose, schema-heavy, hard to stream)
There is no structured output for test execution results that an agent or downstream tool can stream-parse incrementally. Modern test runners all expose this:
- Jest (
--json): https://jestjs.io/docs/cli#--json - pytest-json-report: https://pypi.org/project/pytest-json-report/
- Vitest (JSON reporter): https://vitest.dev/guide/cli.html#reporters
- Bun test (
--reporter=junitand JSON variants)
For LLM/agent consumption, NDJSON is far superior to TRX:
- One event per line: agents can grep/tail without parsing
- Stable schema: agents can rely on field names
- Streamable: events flush as tests complete, enabling live reactions
Proposed shape
Each line is a single JSON object. Event types:
{"event":"run_started","schemaVersion":1,"runId":"...","startTimeUtc":"...","framework":{"name":"MSTest","version":"3.x"}}
{"event":"test_started","testId":"...","testUid":"...","displayName":"...","fullyQualifiedName":"...","source":"..."}
{"event":"test_completed","testId":"...","outcome":"passed|failed|skipped|inconclusive","durationMs":12.3,"messages":[...],"errors":[{"type":"...","message":"...","stackTrace":"..."}],"output":{"stdout":"...","stderr":"..."},"location":{"file":"...","line":42}}
{"event":"run_summary","schemaVersion":1,"runId":"...","durationMs":1234,"counts":{"total":42,"passed":40,"failed":1,"skipped":1,"inconclusive":0},"endTimeUtc":"..."}
Per-test fields are optional and omitted when empty to reduce file size / token count.
CLI surface
--report-json [filename] # writes NDJSON file under results-directory
--report-json-filename <path> # explicit filename
Mirrors --report-trx* / --report-html* exactly.
Implementation outline
- New project:
src/Platform/Microsoft.Testing.Extensions.JsonReport/Microsoft.Testing.Extensions.JsonReport.csproj - Class
JsonReportDataConsumer : IDataConsumerconsumesTestNodeUpdateMessagefrom the message bus (same pattern asTrxDataConsumer). JsonReportCommandLineOptionsProvider : ICommandLineOptionsProviderregisters the two options.- Use
System.Text.JsonUtf8JsonWriterfor AOT-safety and zero-allocation writes. - Document schema in
PACKAGE.mdand adocs/JsonReportSchema.mdpage. - Add
--list-tests jsonwas already added in #8280; this extension is for execution results.
Risks / questions
- Naming:
JsonReportvsJsonlReportvsEventStreamReport. NDJSON is the format; "JsonReport" matches the existing extension-name pattern (Trx, Html, AzureDevOps). - Should the schema also produce one big JSON document (Jest-style
--json) in addition to NDJSON? NDJSON is strictly more useful for streaming agents; the umbrella issue can debate. - Stability: bump
schemaVersionon breaking changes.
Acceptance criteria
- New package
Microsoft.Testing.Extensions.JsonReportproduced and shipped. --report-jsonand--report-json-filenameoptions registered and visible in--helpwith-extensions snapshot.- Documented schema in
PACKAGE.md+ a versioned schema doc. - Acceptance integration tests: run a known failing+passing test set, assert the NDJSON file has the expected events.
References
- Umbrella: #8824
- Existing patterns:
src/Platform/Microsoft.Testing.Extensions.TrxReport/,Microsoft.Testing.Extensions.HtmlReport/,Microsoft.Testing.Extensions.AzureDevOpsReport/ - Related: #8280 (
--list-tests json) - External: Jest
--json, pytest-json-report, Vitest JSON reporter
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the existing extensions in src/Platform/Microsoft.Testing.Extensions.TrxReport/, Microsoft.Testing.Extensions.HtmlReport/, and Microsoft.Testing.Extensions.AzureDevOpsReport/. Define the JsonReport project, data consumer, command-line options, package and schema documentation, then add acceptance integration tests covering passing and failing tests. Done means the package is shipped, both options appear in the help snapshot, the schema is documented, and expected NDJSON events are produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100