Clarify spec intent: common xtra field names, labels vs xtra.traits, stdout/stderr line-splitting
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 93
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Context
We are adding a CTRF reporter to Microsoft.Testing.Platform (the engine behind MSTest, xunit.v3, NUnit, and various other .NET test frameworks). Brad Wilson (xunit) recently published a side-by-side comparison (microsoft/testfx#8903) of his xunit.v3 CTRF emitter vs ours, and several differences boiled down to "the spec doesn't say". Before each ecosystem settles on its own conventions, we would love your guidance on the following so that CTRF documents produced by different .NET reporters are interoperable.
1. Canonical key names for common extra fields
The spec marks environment and per-test as additionalProperties: false for everything except extra, which is the freeform escape hatch. That works, but means two producers reporting the same concept (machine name, exception type, etc.) can pick different keys inside extra, hurting tooling that wants to consume both.
Two concrete examples we hit:
- Machine / hostname: we emit
environment.extra.machine; xunit.v3 emitsenvironment.extra.computer. - Exception type name: we emit
tests[].extra.exceptionType(e.g.System.InvalidOperationException); xunit.v3 emitstests[].extra.exception.
Are there "recommended" key names for these (and other widely useful fields like build agent, runtime version, framework name)? It would be great either to (a) bless one of the two existing names per concept, or (b) publish a non-normative "common extra keys" appendix so future producers can converge.
2. What is labels for?
The spec says labels is "Structured key-value metadata for the test case (e.g. priority, severity, external identifiers)" typed as an object whose values are string | number | boolean.
We read the e.g. as non-exhaustive and use labels for the test framework's user-defined key/value traits (e.g. Category=Smoke, Owner=alice). xunit.v3 instead puts traits under tests[].extra.traits because Brad reads labels as restricted to the three listed examples (priority/severity/external IDs).
Could you clarify the intent?
- Option A -
labelsis the right place for any structured key/value test metadata (our reading). - Option B -
labelsis reserved for a narrow administrative set; arbitrary trait-style metadata should go underextra. - Option C - something else (e.g.
parametersfor data-driven inputs,labelsonly for the three listed examples,extra.traitsfor everything else).
We are happy to follow whichever interpretation matches your intent - just want to avoid two .NET CTRF producers diverging on something this central.
3. Are stdout / stderr meant to be one entry per physical line?
The schema describes both as "Standard output lines from test execution", typed as an array of strings. We initially emitted a single multi-line string (one array element with embedded \n); we have since switched to one element per LF-terminated line (with CRLF normalization and the trailing newline not producing an extra empty entry).
Two follow-ups:
- Is that one-element-per-physical-line interpretation what the spec author intended?
- For tests that produce no captured output, should the array be present-and-empty (
[]), or omitted entirely? We currently omit it; the spec doesn't say.
Thanks for the work on CTRF - it has made a real difference for us standardizing across .NET test frameworks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the CTRF schema descriptions for extra, labels, stdout, and stderr, then compare the linked xunit.v3 and Microsoft.Testing.Platform interpretations. Done means the specification clearly defines common extra keys, labels versus traits, line splitting, and whether empty output arrays are present or omitted.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, testing-qa
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100