ESCOMP / ESCOMP/CTSM

Add a checker for the testlist to verify that the tests make sense

Open
#3,049 4 comments 0 reactions 0 assignees View on GitHub
enhancement testing
Dominant language
Fortran
Stars
352
Forks
361
Avg merge
2d 21h
Merged PRs (30d)
7

Description

One of the reasons for problems in our testlist like #3048 is that we don't have a checker to verify that the testlist doesn't have common problems. #2979 is an example of a checker, but it just checks that syntax is correct. Which is a great thing to check. But, it would be helpful to have something that goes beyond that and does other checking for test validity.

The way we have managed the testlist is with manual examination, and trying to make sure the list makes sense. Which is hard to do with a long list. And it takes a lot of focused effort to do. So it's not something we can do very often.

Ideas for things to check:

1. For example, one of the things checked for could be length by testmod to prevent #3048 with newly added tests.
2. Have it verify which testmods are used in the testlist -- show which ones aren't tested
3. Have it check the types of tests being used and show which ones aren't being used.
4. Have it show the balance between different tests. query_testlist shows by machine/compiler, but also sorting by things like this: resolution, DEBUG or not, test-type, mpi-serial or not, threaded tests, and test-length categories

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.