ctrf-io / ctrf-io/ctrf

Clarify spec intent: common xtra field names, labels vs xtra.traits, stdout/stderr line-splitting

Open
#53 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
93
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Context

We are adding a CTRF reporter to Microsoft.Testing.Platform (the engine behind MSTest, xunit.v3, NUnit, and various other .NET test frameworks). Brad Wilson (xunit) recently published a side-by-side comparison (microsoft/testfx#8903) of his xunit.v3 CTRF emitter vs ours, and several differences boiled down to "the spec doesn't say". Before each ecosystem settles on its own conventions, we would love your guidance on the following so that CTRF documents produced by different .NET reporters are interoperable.


1. Canonical key names for common extra fields

The spec marks environment and per-test as additionalProperties: false for everything except extra, which is the freeform escape hatch. That works, but means two producers reporting the same concept (machine name, exception type, etc.) can pick different keys inside extra, hurting tooling that wants to consume both.

Two concrete examples we hit:

  • Machine / hostname: we emit environment.extra.machine; xunit.v3 emits environment.extra.computer.
  • Exception type name: we emit tests[].extra.exceptionType (e.g. System.InvalidOperationException); xunit.v3 emits tests[].extra.exception.

Are there "recommended" key names for these (and other widely useful fields like build agent, runtime version, framework name)? It would be great either to (a) bless one of the two existing names per concept, or (b) publish a non-normative "common extra keys" appendix so future producers can converge.

2. What is labels for?

The spec says labels is "Structured key-value metadata for the test case (e.g. priority, severity, external identifiers)" typed as an object whose values are string | number | boolean.

We read the e.g. as non-exhaustive and use labels for the test framework's user-defined key/value traits (e.g. Category=Smoke, Owner=alice). xunit.v3 instead puts traits under tests[].extra.traits because Brad reads labels as restricted to the three listed examples (priority/severity/external IDs).

Could you clarify the intent?

  • Option A - labels is the right place for any structured key/value test metadata (our reading).
  • Option B - labels is reserved for a narrow administrative set; arbitrary trait-style metadata should go under extra.
  • Option C - something else (e.g. parameters for data-driven inputs, labels only for the three listed examples, extra.traits for everything else).

We are happy to follow whichever interpretation matches your intent - just want to avoid two .NET CTRF producers diverging on something this central.

3. Are stdout / stderr meant to be one entry per physical line?

The schema describes both as "Standard output lines from test execution", typed as an array of strings. We initially emitted a single multi-line string (one array element with embedded \n); we have since switched to one element per LF-terminated line (with CRLF normalization and the trailing newline not producing an extra empty entry).

Two follow-ups:

  • Is that one-element-per-physical-line interpretation what the spec author intended?
  • For tests that produce no captured output, should the array be present-and-empty ([]), or omitted entirely? We currently omit it; the spec doesn't say.

Thanks for the work on CTRF - it has made a real difference for us standardizing across .NET test frameworks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the CTRF schema descriptions for extra, labels, stdout, and stderr, then compare the linked xunit.v3 and Microsoft.Testing.Platform interpretations. Done means the specification clearly defines common extra keys, labels versus traits, line splitting, and whether empty output arrays are present or omitted.

Written by the indexing model from the issue text.

Assessment

Domain
documentation, testing-qa
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.