catchorg / catchorg/Catch2

Approval tests need cleanup

Open
#2,090 1 comment 0 reactions 0 assignees View on GitHub
Development
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

As the number of tests, first party reporters, and normalization regexes increases, the runtime of ApprovalTests (running `tools/scripts/approvalTests.py` on the `SelfTest` binary) also increases, and it has gotten to a point where the runtime is becoming significant. Thus we should clean up the approval tests and improve their performance.

I see 2 possible approaches:
- [ ] Reduce the number of tests which are run as part of ApprovalTests. We already prevent _some_ tests from being run because their output is platform-specific in a manner that is very annoying to sanitize, so we can just disable more tests from being run during approval tests. (They would still be run as part of unit tests, and lot of tests are very similar in which part of Catch2 they exercise during output)
- [x] Make the normalization smarter. Some of the normalization checks are specific to just one or two reporters, but they are done on outputs of all reporters. By restricting which normalization checks are done on which reporters outputs, and other improvements to their usage, the performance of ApprovalTests would likely be improved significantly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.