Introduce compile-time benchmark suite
- Dominant language
- C++
- Stars
- 21.5k
- Forks
- 3.5k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 2
Description
One of the big advantages of v3 is the improved compilation times, as of preview-3, especially for small testing TUs. The difference between just including v2 release and the basic v3 header, `catch_test_macros.hpp`, is about 80% improvement in throughput. However, these tests were done as one-off compilation tests, rather than being done as a simply runnable test suite.
We should have a real compilation benchmark suite, together with some automation to make running it easy-ish.
Here are some ideas on tests to include:
## Synthetic tests
Synthetic tests are made up tests that do not reflect real world usage, but instead stress some specific feature. These tests should include at least
* [ ] Including the basic header, without any extra code (This is the inclusion overhead in simple case)
* [ ] Small test TU (think 1 test, 2 sections, 3-4 requires)
* [ ] Test TU with a lot of `TEST_CASE`s
* [ ] Test TU with a few `TEST_CASE`s, but each has many `SECTION`s
* [ ] Test TU with a few `TEST_CASE`s, and few `SECTION`s, but with many `REQUIRE`s.
* [ ] Test TU with a few `TEST_CASE`s, `SECTION`s, and `REQUIRE`s, but a lot of `GENERATE`s with same type
* [ ] Test TU with a few `TEST_CASE`s, `SECTION`s, and `REQUIRE`s, but a lot of `GENERATE`s with different type
* [ ] ...
There are many, many, _many_ more possible tests to do, but this would be a very good baseline
## Real world tests
The other direction to take is to pick real world projects using Catch2, modify them so they are compilable with different versions of Catch2, and then measure the compile-time impact on those. The advantage of this approach is that we see impact on projects that actually use Catch2, the disadvantage is that it makes it harder to show that e.g. `SECTION`s improved while `REQUIRE`s regressed. It can also completely overlook impact on less used features (`TEMPLATE_TEST_CASE`, `REQUIRE_THROWS_MATCHES`, etc).
Some possible projects
* [ ] vcpkg
* [ ] Catch2 itself
* [ ] ...
## Other consideration
The heft of C++ headers can change significantly across different language versions, so language version used when compiling should be taken into account as one of the dimensions on results. Same goes for different standard libraries, and different compiler (versions).
Contributor guide
Assessment
This issue has not been assessed yet.