ESMCI / ESMCI/cime

TestStatus: Still would like clearer distinction between test comparisons and baseline comparisons

Open
#396 11 comments 0 reactions 1 assignee Claimed by @jgfouca View on GitHub
Low Priority
Dominant language
Python
Stars
174
Forks
225
Avg merge
1d 16h
Merged PRs (30d)
14

Description

Inspired by @gold2718 's comment here:

https://github.com/ESMCI/cime/pull/392#issuecomment-239626725
- where he suggests knowing ahead of time which test phases are expected: I realized that it would still be good to have a clearer distinction made between history file comparisons done as an integral part of a test (e.g., comparing .base with .rest files for an ERS test) vs. baseline comparisons. Even though these are both about comparing history files, they are fundamentally different in their meaning.

cime4 test reporting made a clear distinction between these, as:

```
--- Test Functionality ---:
PASS ERS_D_Ld10.T31_g37_gl5.IGHISTCLM45.yellowstone_intel.clm-glcMEC_decrease.clm2.h0.nc : test compare clm2.h0 (.base and .rest files)

--- Baseline Comparison ---:
PASS ERS_D_Ld10.T31_g37_gl5.IGHISTCLM45.yellowstone_intel.clm-glcMEC_decrease.cism.h.nc : baseline compare cism.h (baseline: compare .base file with clm4_5_9_r185 file)
```

and I feel it would be good to maintain that clear distinction in cime5.

@jgfouca made some steps towards this, by adding suffixes to the single 'COMPARE' phase. But it seems like it would be good to explicitly split this into two phases - a TEST_COMPARE phase and a BASELINE_COMPARE phase - both in the code and in the test reporting. In the test reporting, this will make it easier to do greps to include / exclude particular aspects of the tests. In the code, this will make it easier to add new functionality, such as the functionality proposed by @gold2718 .

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.