ESMCI / ESMCI/cime

Add additional ways of counting tests to query_testlist

Open
#4,776 6 comments 0 reactions 0 assignees View on GitHub
Low Priority Responsibility: CESM Responsibility: CTSM topic: tests ty: enhancement
Dominant language
Python
Stars
174
Forks
225
Avg merge
1d 16h
Merged PRs (30d)
14

Description

We are finding that it's hard to balance long testlists and check what coverage is going on in them. We've been doing this kind of thing by hand, but it's hard to do well manually. It takes a lot of hard focussed manual work to get the picture of all the tests and make sure the coverage makes sense. And really what needs to be done is to sort the tests in different ways. So you have to put it into a spreadsheet or something. Since this is such a manual process it's not something we can do very often.

scripts/query_testlists has this option:

> --count Rather than listing tests, just give counts by category/machine/compiler.

This is helpful, but we'd like to see it extended to count other things as well.

This is a list of things to count that we think would be helpful in CTSM:

- resolution
- testmods
- test-types (e.g., ERS, SMS etc.)
- Specific test options:
- _D (and without _D)
- _M options (_Mmpi-serial, _Mopenmpi, or _M not used etc.)
- _P PE layouts of tests (and without _P)
- _L test-length categories (and without _L)

So for example using this now in CTSM I see this:

``` shell
./query_testlists --xml-category aux_clm --count
aux_clm: 323
derecho: 275
gnu: 80
intel: 192
nvhpc: 3
izumi: 48
intel: 1
nag: 47
```

I'd like it to add lines something like:

```
aux_clm: 323
derecho: 275
Compilers:
gnu: 80
intel: 192
nvhpc: 3

Broken down by test type for all compilers:

Resolutions for all compilers: 275
f09_g17: 5
f10_f10_mg37: 150
f19_g17: 9
ne30pg3_ne30pg3_mg17: 6
.
.
.
Build options: 275
_D DEBUG tests: 175
(without _D) production tests: 100
Test types: 275
ERI 20
ERS 70
ERP 50
SMS 100
.
.
.
Testmods: 275
default. 50
clm60cam7LndTuningMode 10
.
.
.
(no testmod) 10
MPI options: 275
_Mmpi-serial: 50
(without _M): 225
PE Layouts: 275
_P128x1: 50
_P256x2: 20
.
.
.
(without a _P option): 100
Test lengths: 275
_Ld65: 30
_Ld365: 10
.
.
.
(without a _L option): 100

```

So each category is for all compilers for the given machine. It also lists the number of tests counted (which should agree with the total number of tests for the machine), and the count for each specific option. And the sum of the tests under the option should add up to the total for that option.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.