Approval tests need cleanup
- Dominant language
- C++
- Stars
- 21.5k
- Forks
- 3.5k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 2
Description
As the number of tests, first party reporters, and normalization regexes increases, the runtime of ApprovalTests (running `tools/scripts/approvalTests.py` on the `SelfTest` binary) also increases, and it has gotten to a point where the runtime is becoming significant. Thus we should clean up the approval tests and improve their performance.
I see 2 possible approaches:
- [ ] Reduce the number of tests which are run as part of ApprovalTests. We already prevent _some_ tests from being run because their output is platform-specific in a manner that is very annoying to sanitize, so we can just disable more tests from being run during approval tests. (They would still be run as part of unit tests, and lot of tests are very similar in which part of Catch2 they exercise during output)
- [x] Make the normalization smarter. Some of the normalization checks are specific to just one or two reporters, but they are done on outputs of all reporters. By restricting which normalization checks are done on which reporters outputs, and other improvements to their usage, the performance of ApprovalTests would likely be improved significantly.
Contributor guide
Assessment
This issue has not been assessed yet.