GENERATE doesn't display which values caused test failures in the test report
- Dominant language
- C++
- Stars
- 21.5k
- Forks
- 3.5k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 2
Description
**Description**
The GENERATE macro runs a given test multiple times with different parameters, but the test report doesn't seem to provide context or information on which of those parameters are causing failures in the test.
**Additional context**
As an example, let's consider the following simple test:
```
TEST_CASE("Test with generators")
{
int value = GENERATE(1, 3, 5);
REQUIRE_FALSE(testFunction(value));
}
```
If the test fails with generated parameter values 1 and 3, this is what the test report looks like:
```
-------------------------------------------------------------------------------
Test with generators
-------------------------------------------------------------------------------
C:\TestProject1\Tests1.cpp(131)
...............................................................................
C:\TestProject1\Tests1.cpp(134): FAILED:
REQUIRE_FALSE( testFunction(value) )
with expansion:
!true
-------------------------------------------------------------------------------
Test with generators
-------------------------------------------------------------------------------
C:\TestProject1\Tests1.cpp(131)
...............................................................................
C:\TestProject1\Tests1.cpp(134): FAILED:
REQUIRE_FALSE( testFunction(value) )
with expansion:
!true
-------------------------------------------------------------------------------
```
As we can see, there aren't any hints whatsoever that the values 1 and 3 were the reasons of those failures, while 5 was good.
If there isn't a way to have Catch2 display the generated parameters on the report, is this something that would be feasible to add?
Are there any reasons why this information doesn't show up in the test report?
Contributor guide
Research direction
Start by tracing the GENERATE macro through Catch2's test execution and reporting paths, then inspect existing tests for generated cases and failure output. Done means failed generated cases identify the parameter values that caused each failure while preserving the existing report behavior for passing cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100