Test suite redesign: consider having a subset of the test suite for baseline comparisons
- Dominant language
- Fortran
- Stars
- 352
- Forks
- 361
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 7
Description
Broadly speaking, the system test suite serves two purposes:
1. To make sure that various requirements are met - the code compiles and runs on various machines and compilers, passes exact restart tests, etc.
2. To ensure that changes that are expected to be bit-for-bit are indeed bit-for-bit.
We've been using the same test suite for both purposes, but they have somewhat different characteristics: (1) requires a broader variety of tests (including various machines/compilers, specialty test types like NCK, etc.) but has the benefit that you can run it less frequently; (2) can be done with a somewhat more limited set of tests, but I often want to do this testing more frequently: in the course of making an answer-changing tag, I often go back and forth between answer-preserving and answer-changing modifications, and I want to be sure that my answer-preserving modifications truly don't change answers (currently I run the full test suite to ensure this, but that takes 4-6 hours).
Before bringing changes to master, it's important that all tests (covering both (1) and (2)) be run. But for intermediate testing during branch development, I'd find it helpful if we created a separate test suite that is significantly smaller than the full test suite (e.g., 1/2 the size of the full test suite or less), which has just enough coverage in terms of baseline comparisons that it could give me confidence that my most recent changes don't change answers.
One helpful piece of this would be if all important baseline tests were on a single machine. This simplifies the process of running this smaller, frequently-run test suite, and could also simplify the process of the full test suite (i.e., if we just need to run baseline comparison and generation on a single machine; or, even if we do generally run it on multiple machines, it's not a huge deal if the other machine(s) are down, and we can do just a cursory look at results for the other machine(s)). This is the process I use for CISM testing, and I find it streamlines the process of making a tag.
In addition, this test suite can exclude "specialty" tests like NCK and LII, as long as the configuration they're testing is covered by some other test: for the sake of baseline comparisons, pretty much all test types are equivalent.
We could accomplish this by having two separate test categories:
- `aux_clm_baselines`: SMS, ERS and ERP tests on the primary test machine (using a mix of compilers available on that machine)
- `aux_clm_???`: Other test types, a second machine, etc.
@ekluzek @bandre-ucar any thoughts on this?
Contributor guide
Assessment
This issue has not been assessed yet.