microsoft / microsoft/testfx

Add JSON / NDJSON test report extension (Microsoft.Testing.Extensions.JsonReport)

Open
#8,828 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/mtp area/mtp-reporting
Dominant language
C#
Stars
1k
Forks
312
Avg merge
8h 30m
Merged PRs (30d)
469

Description

Summary

Introduce a new MTP extension Microsoft.Testing.Extensions.JsonReport that emits a stable, documented NDJSON (newline-delimited JSON) event stream of test execution events, plus a final summary event. Models the existing Microsoft.Testing.Extensions.TrxReport, Microsoft.Testing.Extensions.HtmlReport, and Microsoft.Testing.Extensions.AzureDevOpsReport extensions.

Motivation

Today MTP offers two machine-readable outputs:

  • --list-tests json (test discovery only, added in #8280)
  • TRX XML via Microsoft.Testing.Extensions.TrxReport (verbose, schema-heavy, hard to stream)

There is no structured output for test execution results that an agent or downstream tool can stream-parse incrementally. Modern test runners all expose this:

For LLM/agent consumption, NDJSON is far superior to TRX:

  • One event per line: agents can grep/tail without parsing
  • Stable schema: agents can rely on field names
  • Streamable: events flush as tests complete, enabling live reactions

Proposed shape

Each line is a single JSON object. Event types:

{"event":"run_started","schemaVersion":1,"runId":"...","startTimeUtc":"...","framework":{"name":"MSTest","version":"3.x"}}
{"event":"test_started","testId":"...","testUid":"...","displayName":"...","fullyQualifiedName":"...","source":"..."}
{"event":"test_completed","testId":"...","outcome":"passed|failed|skipped|inconclusive","durationMs":12.3,"messages":[...],"errors":[{"type":"...","message":"...","stackTrace":"..."}],"output":{"stdout":"...","stderr":"..."},"location":{"file":"...","line":42}}
{"event":"run_summary","schemaVersion":1,"runId":"...","durationMs":1234,"counts":{"total":42,"passed":40,"failed":1,"skipped":1,"inconclusive":0},"endTimeUtc":"..."}

Per-test fields are optional and omitted when empty to reduce file size / token count.

CLI surface

--report-json [filename]      # writes NDJSON file under results-directory
--report-json-filename <path> # explicit filename

Mirrors --report-trx* / --report-html* exactly.

Implementation outline

  • New project: src/Platform/Microsoft.Testing.Extensions.JsonReport/Microsoft.Testing.Extensions.JsonReport.csproj
  • Class JsonReportDataConsumer : IDataConsumer consumes TestNodeUpdateMessage from the message bus (same pattern as TrxDataConsumer).
  • JsonReportCommandLineOptionsProvider : ICommandLineOptionsProvider registers the two options.
  • Use System.Text.Json Utf8JsonWriter for AOT-safety and zero-allocation writes.
  • Document schema in PACKAGE.md and a docs/JsonReportSchema.md page.
  • Add --list-tests json was already added in #8280; this extension is for execution results.

Risks / questions

  • Naming: JsonReport vs JsonlReport vs EventStreamReport. NDJSON is the format; "JsonReport" matches the existing extension-name pattern (Trx, Html, AzureDevOps).
  • Should the schema also produce one big JSON document (Jest-style --json) in addition to NDJSON? NDJSON is strictly more useful for streaming agents; the umbrella issue can debate.
  • Stability: bump schemaVersion on breaking changes.

Acceptance criteria

  • New package Microsoft.Testing.Extensions.JsonReport produced and shipped.
  • --report-json and --report-json-filename options registered and visible in --help with-extensions snapshot.
  • Documented schema in PACKAGE.md + a versioned schema doc.
  • Acceptance integration tests: run a known failing+passing test set, assert the NDJSON file has the expected events.

References

  • Umbrella: #8824
  • Existing patterns: src/Platform/Microsoft.Testing.Extensions.TrxReport/, Microsoft.Testing.Extensions.HtmlReport/, Microsoft.Testing.Extensions.AzureDevOpsReport/
  • Related: #8280 (--list-tests json)
  • External: Jest --json, pytest-json-report, Vitest JSON reporter

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the existing extensions in src/Platform/Microsoft.Testing.Extensions.TrxReport/, Microsoft.Testing.Extensions.HtmlReport/, and Microsoft.Testing.Extensions.AzureDevOpsReport/. Define the JsonReport project, data consumer, command-line options, package and schema documentation, then add acceptance integration tests covering passing and failing tests. Done means the package is shipped, both options appear in the help snapshot, the schema is documented, and expected NDJSON events are produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.