catchorg / catchorg/Catch2

Junit Reporter breaks when BDD Sections fail

Open
#1,650 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

**Describe the bug**
When a failure occurs, all previously-occurring tests within the same top-level section have their names messed up in a particular way. For example, the following test:

```
SCENARIO("This is a fake test", "[fake]"){
GIVEN("Prior given"){
REQUIRE(true);
THEN("Prior Given Inner"){
REQUIRE(true);
}
}
GIVEN("Given1"){
THEN("Beep Boop"){
REQUIRE(true);
}
WHEN("When1"){
THEN("Success"){
REQUIRE(true == true);
}
THEN("PRIOR WITH INNER"){
AND_THEN("inner prior"){
REQUIRE(true);
}
}
THEN("FAIL"){
AND_THEN("SUCCEED INNER"){
REQUIRE(true);
}
AND_THEN("FAIL INNER") {
FAIL("FAILURE");
}
}
THEN("THEN AFTER FAILURE"){
AND_THEN("inner after") {
REQUIRE(true);
}
}
THEN("LAST"){
REQUIRE(true);
}
}
}
GIVEN("Given 2"){
REQUIRE(true);
}
}
THEN("FAIL"){
AND_THEN("FAIL INNER") {
FAIL("FAILURE");
}
}
THEN("THEN AFTER FAILURE"){
AND_THEN("inner after") {
REQUIRE(true);
}
}
THEN("LAST"){
REQUIRE(true);
}
}
}
GIVEN("Given 2"){
REQUIRE(true);
}
}
```

Produces the following junit xml output:

```

FAILURE at /cygdrive/c/Users/dbak/Projects/firmware/unit-tests/FlaAlgoTests/flamenco_algorithms_anom_stict_tuning_tests.cpp:111

```
So everything after the failure is fine, and everything before the root of the failure is okay. But among all of the leafs occurring before and including the failing leaf, there seem to be *lots* of problems. All of them have the failing leaf's description prepended to their path, but then their own path seems to be reversed or jumbled up - I can't figure out a pattern to it.

This is a big problem for us as it might mean we need to move away from BDD-style tests and change our existing ones - all of our test reports have become totally unreadable because of this.

**Platform information:**
- OS: Windows 10
- Compiler+version: g++
- Catch version: 2.8.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.