catchorg / catchorg/Catch2

Custom CulmulativeReporterBase receives odd children order when dealing with Skipped tests

Open
#2,928 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

**Describe the bug**
I'm building a reporter from CulmulativeReporterBase. I'm iterating the children of the m_testRun, and walking through each child SectionNode.

```
void LogSectionNode(const Catch::CumulativeReporterBase::SectionNode& node, const std::string& prefix)
{
std::cout << prefix << "Section: " << node.stats.sectionInfo.name <<
" (child count: " << node.childSections.size() << " passed:" << node.stats.assertions.passed << " skipped:" << node.stats.assertions.skipped << ")\n";
for(const auto& childSection : node.childSections)
{
LogSectionNode(*childSection, prefix + " ");
}
}

void ReproReporter::testRunEndedCumulative()
{
for (const auto& childNode : m_testRun->children)
{
std::cout << "test node: " << childNode->value.testInfo->name << '\n';
for (const auto& childSection : childNode->children)
{
LogSectionNode(*childSection, " ");
}
}
}
```

I'm running it against a Scenario such as this:

```
SCENARIO("Repro")
{
GIVEN("ReproGiven")
{
WHEN("ReproWhenA")
{
THEN("ReproThenA1")
{
SUCCEED();
}
THEN("ReproThenA2")
{
SUCCEED();
}
}
WHEN("ReproWhenB")
{
//SKIP("skip");
SUCCEED();
}
}
}
```

When I uncomment the skip, I get very odd output:

```
test node: Scenario: Repro
Section: Scenario: Repro (child count: 2 passed:0 skipped:0)
Section: When: ReproWhenB (child count: 2 passed:0 skipped:1)
Section: When: ReproWhenA (child count: 2 passed:1 skipped:0)
Section: Then: ReproThenA1 (child count: 0 passed:1 skipped:0)
Section: Then: ReproThenA2 (child count: 0 passed:1 skipped:0)
Section: Given: ReproGiven (child count: 0 passed:0 skipped:1)
Section: Given: ReproGiven (child count: 0 passed:0 skipped:0)
```
Note that the scenario has two children, once called ReproWhenB and another called ReproGiven. There's a second ReproGIven under ReproWhenB, that contains the actual skipped test.

**Expected behavior**
Compare that to the output when I don't skip:

```
test node: Scenario: Repro
Section: Scenario: Repro (child count: 1 passed:1 skipped:0)
Section: Given: ReproGiven (child count: 2 passed:1 skipped:0)
Section: When: ReproWhenA (child count: 2 passed:1 skipped:0)
Section: Then: ReproThenA1 (child count: 0 passed:1 skipped:0)
Section: Then: ReproThenA2 (child count: 0 passed:1 skipped:0)
Section: When: ReproWhenB (child count: 0 passed:1 skipped:0)
```
Now the scenario has one child, ReproGIven, which has two children, WhenA and WhenB. Nothing here looks weird.

**Reproduction steps**
https://github.com/tenpn/catch2-skipreporter-repro
download, build with `xbuild build`, comment/uncomment the skip in src/ReproScenario.cpp, run the tests with `xbuild run binary -r repro`.

**Platform information:**

- OS: **Windows 10**
- Compiler+version: **VS 2022 Community**
- Catch version: **v3.7.0**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.