Add a checker for the testlist to verify that the tests make sense
- Dominant language
- Fortran
- Stars
- 352
- Forks
- 361
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 7
Description
One of the reasons for problems in our testlist like #3048 is that we don't have a checker to verify that the testlist doesn't have common problems. #2979 is an example of a checker, but it just checks that syntax is correct. Which is a great thing to check. But, it would be helpful to have something that goes beyond that and does other checking for test validity.
The way we have managed the testlist is with manual examination, and trying to make sure the list makes sense. Which is hard to do with a long list. And it takes a lot of focused effort to do. So it's not something we can do very often.
Ideas for things to check:
1. For example, one of the things checked for could be length by testmod to prevent #3048 with newly added tests.
2. Have it verify which testmods are used in the testlist -- show which ones aren't tested
3. Have it check the types of tests being used and show which ones aren't being used.
4. Have it show the balance between different tests. query_testlist shows by machine/compiler, but also sorting by things like this: resolution, DEBUG or not, test-type, mpi-serial or not, threaded tests, and test-length categories
Contributor guide
Assessment
This issue has not been assessed yet.