Seperate `REQUIRE` (total show-stopper) for checking unit-test integrity
- Dominant language
- C++
- Stars
- 21.5k
- Forks
- 3.5k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 2
Description
Sometimes unit tests need to perform their own calculations to compare against those of the tested code. For example, in my use-case, I check that an optimised simulator is producing the same results as a slow direct mathematical evaluation.
This often requires some quick checks of the internal testing code, to check pre- and post-conditions of the helper functions, to help distinguish a bad test from an error in the tested code. This is especially helpful for external contributors writing their own unit tests which use the helper funcitons.
Right now, I use `REQUIRE` in the testing, and in checking the integrity of the internal code. Here's an example; a unit-test which performs its own matrix calculations in order to replicate a tested code's result (which is computed in an entirely different way), and utilises internal testing code:
```C++
vector> getReferenceSum(vector> a, vector> b) {
REQUIRE( a.size() == b.size() );
vector> s = a;
for (size_t r=0; r> a = ...
vector> b = ....
vector> librarySum = superAwesomeFancyFunc(a, b);
vector> referenceSum = getReferenceSum(a, b);
REQUIRE( librarySum == referenceSum );
}
```
This means though that in the ultimately performed unit-test, the checking of pre & post-conditions of the helper functions is included in the statistics of the unit-tests (total number of assertions passed).
Instead, it would be great if there was a seperate "meta" macro, which asserts conditions the test-author believes is tautological, and which when failed (indicating bad unit-testing), stops all testing.
E.g. something like
```C++
DEMAND( cond )
```
Is there currently any way to do this, or at least to seperate the statistics of 'internal' `REQUIRE`s from the 'testing' `REQUIRE`s?
Otherwise, is there a way to write custom macros and 'hook' them into Catch's control-flow (rather than just having them force-exit)?
Contributor guide
Research direction
The issue names no repository files or tests. Start by reviewing Catch2's existing REQUIRE implementation and assertion statistics/control-flow hooks; done would require a documented design and implementation for a separate meta assertion mechanism or an equivalent supported way to exclude internal checks from test statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100