microsoft / microsoft/testfx

What is the intended purpose for the hierarchy provided in TestNode?

Open
#2,537 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/mtp
Dominant language
C#
Stars
1k
Forks
312
Avg merge
8h 30m
Merged PRs (30d)
469

Description

When reporting TestNode updates, you can include the parent test node UID. Looking at the code in Microsoft.Testing.Extensions.VSTestBridge for examples, it doesn't appear that any reporting there is using any hierarchy.

What do you envision the purpose of that hierarchy is, and how would it be reported to the user in Test Explorer? xUnit.net, for example, can make a case for its hierarchy being Assembly > Test Collection > Test Class > Test Method > Test Case > Test. Would it make sense for us to report all of that? Some of that? None of that?

Experimentation with dotnet test today shows that none of that appears to be used at the moment. Reporting parentless test results, just like the VSTestBridge, results in successful execution:

  $ .\src\xunit.v3.assert.tests\bin\Debug\net6.0\xunit.v3.assert.tests.exe

xUnit.net v3 In-Process Runner v0.1.1-pre.379-dev+f22e679603 (64-bit .NET 6.0.27)
  Discovering: xunit.v3.assert.tests (method display = ClassAndMethod, method display options = None)
  Discovered:  xunit.v3.assert.tests (1122 test cases to be run)
  Starting:    xunit.v3.assert.tests (parallel test collections = on [24 threads], stop on fail = off, explicit = off, seed = 1016941239, culture = invariant)
  Finished:    xunit.v3.assert.tests
=== TEST EXECUTION SUMMARY ===
   xunit.v3.assert.tests  Total: 1353, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0, Time: 0.163s
  $ dotnet test src\xunit.v3.assert.tests --framework net6.0

  Determining projects to restore...
  All projects are up-to-date for restore.
  xunit.v3.assert -> C:\Dev\xunit\xunit\src\xunit.v3.assert\bin\Debug\net6.0\xunit.v3.assert.dll
  xunit.v3.common -> C:\Dev\xunit\xunit\src\xunit.v3.common\bin\Debug\netstandard2.0\xunit.v3.common.dll
  xunit.v3.core -> C:\Dev\xunit\xunit\src\xunit.v3.core\bin\Debug\netstandard2.0\xunit.v3.core.dll
  xunit.v3.runner.common -> C:\Dev\xunit\xunit\src\xunit.v3.runner.common\bin\Debug\netstandard2.0\xunit.v3.runner.common.dll
  xunit.v3.runner.inproc.console -> C:\Dev\xunit\xunit\src\xunit.v3.runner.inproc.console\bin\Debug\net6.0\xunit.v3.runner.inproc.console.dll
  xunit.v3.runner.utility -> C:\Dev\xunit\xunit\src\xunit.v3.runner.utility\bin\Debug\netstandard2.0\xunit.v3.runner.utility.netstandard20.dll
  xunit.v3.runner.tdnet -> C:\Dev\xunit\xunit\src\xunit.v3.runner.tdnet\bin\Debug\netstandard2.0\xunit.v3.runner.tdnet.dll
  xunit.v3.runner.tdnet -> C:\Dev\xunit\xunit\src\xunit.v3.runner.tdnet\bin\Debug\netstandard2.0\merged\xunit.v3.runner.tdnet.dll
  xunit.v3.assert.tests -> C:\Dev\xunit\xunit\src\xunit.v3.assert.tests\bin\Debug\net6.0\xunit.v3.assert.tests.dll
  Run tests: 'C:\Dev\xunit\xunit\src\xunit.v3.assert.tests\bin\Debug\net6.0\xunit.v3.assert.tests.dll' [net6.0|x64]
  Tests succeeded: 'C:\Dev\xunit\xunit\src\xunit.v3.assert.tests\bin\Debug\net6.0\xunit.v3.assert.tests.dll' [net6.0|x64]

  $ cat src\xunit.v3.assert.tests\bin\Debug\net6.0\TestResults\xunit.v3.assert.tests_net6.0_x64.log

Microsoft(R) Testing Platform Execution Command Line Tool
Version: 1.0.2+1686a9c93 (UTC 2024/02/21)
RuntimeInformation: win10-x64 - .NET 6.0.27
Copyright(c) Microsoft Corporation.  All rights reserved.
  Discovering: xunit.v3.assert.tests (method display = ClassAndMethod, method display options = None)
  Discovered:  xunit.v3.assert.tests (1122 test cases to be run)
  Starting:    xunit.v3.assert.tests (parallel test collections = on [24 threads], stop on fail = off, explicit = off, seed = 1016941239, culture = invariant)
  Finished:    xunit.v3.assert.tests
=== TEST EXECUTION SUMMARY ===
   xunit.v3.assert.tests  Total: 1353, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0, Time: 0.158s
Passed! - Failed: 0, Passed: 1353, Skipped: 0, Total: 1353, Duration: 320ms - xunit.v3.assert.tests.dll (win10-x64 - .NET 6.0.27)

=== COMMAND LINE ===
C:\Program Files\dotnet\dotnet.exe exec C:\Dev\xunit\xunit\src\xunit.v3.assert.tests\bin\Debug\net6.0\xunit.v3.assert.tests.dll --internal-msbuild-node testingplatform.pipe.ae8e4422d11c4007b5802f10ca515653

I have not attempted to add any hierarchy (this is just experimental work at the moment), but I'm not sure how dotnet test would interpret that currently. Let's assume a single level of hierarchy (Assembly > Test) was supported. Would I report the assembly as "in progress", then at the end, report it as "failed" or "passed" based on whether any tests failed? How would that contribute to the test count in dotnet test? The API description of test node includes a node type of action vs. group (it would make sense that everything that's not a running test would be a group), but I don't see that reflected into the TestNode object in Microsoft.Testing.Platform. I'm presuming that every TestNode is implicitly an action node type right now; is that correct? So is it not possible to reporting groupings today?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked TestNode protocol description and the Microsoft.Testing.Extensions.VSTestBridge reporting examples. Reproduce the parentless results with dotnet test, then investigate how action and group nodes, hierarchy, status, and test counts are currently interpreted. Done means a decided model for supported hierarchy and its reporting behavior, documented or implemented with corresponding tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.