Rework CLM test suite: miscellaneous things to do to shorten turnaround time and increase coverage

There are a number of changes we could make to shorten the turnaround time of the test suite while increasing coverage. This issue documents some ideas.

I feel like our goal should be a sub-4-hour test suite – or at the very least, sub-4-hours for the subset of the test suite for which baseline comparisons are important (see a comment below for thoughts on possibly separating this subset of the test suite). Edit (2019-01-11) I'd also like to get the core-hour requirement down to something like 2,000, and ideally even lower.

Here are some ideas:

- [ ] Change a bunch of tests from 1850 to transient: this covers more code (e.g., http://bugs.cgd.ucar.edu/show_bug.cgi?id=2207 slipped through because we didn't test ciso in combination with a transient case)

- [ ] Rework test suite to get better balance between different compilers, to shorten turnaround time. If you fire off a different test suite for each machine/compiler (which is the standard practice), then the ideal is to have roughly equal balance between the different `machine_compiler` combinations, at least for `cheyenne_intel`, `cheyenne_gnu` and `hobart_nag`. (However, gnu is less good at picking up debug issues, so it's not good to have important configurations only covered by gnu debug.)

- [x] Remove or rework tests that take a large number of processor-hours.
- high-res cases should just be short smoke tests, non-debug or very short debug
- VIC tests don't really need to be at f09 (I think they were originally put at f09 because that was the scientifically-supported resolution for VIC, but for software testing, coarse-resolution is sufficient, with maybe just one very short f09 smoke test - e.g., a few time steps)
- we shouldn't have high-resolution ERI tests, since ERI tests are expensive
- However: it does seem important to have at least one long (e.g., 5-year) global restart test: our 5-year ERS test at f10 resolution has often picked up problems that shorter tests miss (I suspect that a somewhat shorter test at higher resolution would do the same job: I think the key is having a test that runs over enough gridcells x timesteps that it covers most code paths for at least the most important configuration(s))
- (2020-09-18) Could probably shorten the 1-degree PFS test to 10 days rather than 20
- (2020-09-30) Can move some non-critical tests to the new "ctsm_sci" test list. e.g., `SMS_Ln9.ne30pg2_ne30pg2_mg17.I2000Clm50BgcCrop.cheyenne_intel.clm-clm50cam6LndTuningMode` takes a long time, due to time needed for init_interp.
- (2022-03-09) Would it help to significantly decrease the processor count for high-resolution tests, at least the short ones? A lot of the time in these tests is spent in model initialization, and we could probably decrease the core-hour cost significantly with minimal impact on turnaround time by decreasing the processor count for these tests. (A better solution is to change these to coarse-resolution tests, but there are some tests that need to be at production resolution.)
- (2022-03-14) Yes, this seems to be a good solution. I did this for a couple of tests in 32448878ba7445b3ac8ca8ad959291a235a0e761 and found that the total test run time is about the same with 1/10 the processors – and in some cases even faster, due to most of the total run time being spent in model initialization.
- (2022-03-25) I have done this pretty well in #1688 . There's probably more that could be done, but we're in a pretty good place now in this respect.

- [x] (2018-10-27) Make more tests use SGLC rather than CISM, to shorten build times
- (2020-11-02) Addressed with #1135, in ctsm5.1.dev011

- [ ] (2019-01-23) Remove / rework some tests that have a long wallclock time, unless the long time is really necessary for the test.
- Example: `SMS_D_Ly2.1x1_numaIA.IHistClm50BgcCropGs.cheyenne_intel.clm-ciso_bombspike1963` (currently takes > 2 hours).
- Rationale: These long tests can delay the completion of the full test suite, unless they start early on. And if they fail due to system issues, it can take a long time to wait for them to rerun.

- [ ] (2019-04-29) Consider changing some multi-year tests to short, decStart tests. In particular: we have at least one or two (and possibly more) multi-year tests because of #404 . With ctsm1.0.dev036, that issue seems to be resolved. We should look critically at our global tests that cross the year boundary and consider if any of them can/should be changed to decStart tests.
- One in particular is the new LCISO test: Erik made that an Lm13 test rather than a short, decStart test because of #404 . We could consider making it a short, decStart test. However, note that the case with ciso would then fail in the wrong way if we tested this test by reintroducing the bug that was fixed in ctsm1.0.dev036 (it would fail with a C balance error rather than the LCISO test failing in the comparison between cases). We should think about whether that's okay, or if that suggests that we should keep this as a 13-month test. My initial inclination is that that's okay, and it's fine to change this to a decStart test. At the very least, though, we should confirm that the decStart LCISO test would still pick up problems by (1) reintroducing the bug that was fixed in ctsm1.0.dev036, and (2) commenting out the endrun associated with the C balance check: confirm that the LCISO test fails the comparison between cases, as expected.

- [x] (2019-05-01) I don't think we're getting much benefit from long single-point tests like `ERS_Ly20_Mmpi-serial.1x1_numaIA.I2000Clm50BgcDvCropQianGs.cheyenne_intel.clm-cropMonthOutput` (and maybe others?), unless they are constructed specially to target particular things (like `ERS_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianGs.cheyenne_intel.clm-cropMonthOutput # restart is right before increasing natural veg to > 0 while also shifting PCT_CFT`). I think that, at one point, these long tests were important to test crop more fully, since most crop tests started from cold start and only tested the first few years of crop, at most. Now that nearly everything uses spun-up initial conditions, this is no longer the case. One way to keep these long single-point tests valuable would be to make them use cold start: then they would be testing the first couple of decades of run from cold start, which isn't otherwise tested.
- (2022-02-24) In #1660 I have changed these long single point tests to be cold start, so at least we're getting some unique benefit from them

- [x] (2020-05-20) Substantially trim the PTS mode testing (ptsRLA, ptsRLB, ptsROA tests). Erik says:
> You are certainly right we have plenty of PTS_MODE testing. And maybe too much. Although it might be good to see what testing CAM does for SCAM and make sure our testing lines up that way. RLA and RLB are just two different points, so we don't need everything to run with both. ROA is a point over ocean to make sure that works when running SCAM. So we probably only need one of those.

Then:

> I just looked at the CAM testing for SCAM and it looks like they just test with cheyenne_intel, and only have three tests for it. Originally I thought PTS_MODE could be useful for CLM developers -- but it hasn't worked out that way. It's much easier in CLM (than in CAM) to just subset the files for a single point rather than having a special configuration for it. So I'd say we could cut back our PTS_MODE testing in CTSM from what it is now.

- (2022-02-24) I have trimmed the PTS mode testing in #1660 . See also this comment below: https://github.com/ESCOMP/CTSM/issues/275#issuecomment-1049331418

- [ ] (2020-06-07) We could change a bunch of our tests to use Qian datm forcing in order to speed up datm, since this likely wouldn't decrease the test coverage of the CTSM code. (However, this is only helpful if datm is a limitation in tests; this is true for single-point cases, but I'm not sure if it's true for global cases. We should check whether datm is a limiting factor in our typical f10 cases and other tests; if so, we should consider making this or a similar change.)

- [x] (2020-09-18) For tests that use a lot of processors: make some changes so they get through the queue faster:
- try to reduce the walltime limit to something pretty small so they get through the queue faster
- consider using an alternative, smaller PE layout for these tests, either by hard-coding the PE layout in the test (e.g., `_P720x1`) or using the mechanism to give multiple PE layouts for a given configuration in config_pes.xml
- (2020-11-04) I changed our C96 test to `P360x2`, reducing its TOTALPES from 3528 to 720. This reduced its queue wait time from hours to seconds. I thought about introducing an alternative PE layout in `config_pes.xml`, but that felt overly complex given that we only need this for testing. (At first I liked that an alternative PE layout could allow us to keep datm on its own processors, but from looking at the test timing, this wouldn't make much, if any, difference.)
- (2021-04-02) I could imagine having all tests in the aux_clm test suite have small PE counts, with production-level PE counts restricted to the clm_sci test list, which we should run every month or two (without baseline comparisons)
- (2022-03-14) I started applying this idea to some tests in 32448878ba7445b3ac8ca8ad959291a235a0e761. This seems like a good approach: the test run time is about the same as before with about 1/10 the number of processors.
- (2020-11-16) Note that, in the latest run of the test suite, everything except the f19 and f09 tests finished, and those higher-resolution tests were still waiting in the queue, highlighting the importance of this.
- Could we add a feature to the test list that lets you specify the queue for a single test? Then we could specify the premium queue for the large-processor-count-but-short tests.
- (2022-03-25) I have done this in #1688 by having all tests run on 5 nodes or fewer.

- [x] (2020-10-06) Single point tests on cheyenne have seemed flaky lately, frequently dying due to system issues. My guess is that this is because they are on shared nodes. Is there something we can do about this? We could move them to full nodes on izumi, or maybe even full nodes on cheyenne if they are short enough... though it tends to be the longer tests that are dying. So maybe the solution is just to get rid of the long single-point tests, as noted above.
- (2022-02-24) This is addressed in #1660

- [ ] (2020-10-19) Make sure that all tests on izumi use reduced output, since i/o is often a huge runtime cost of the izumi tests.

- [x] (2020-11-04) Consider making all izumi tests single-node, since multi-node tests may be more prone to system issues.

- [ ] (2021-03-29) Consider setting the co2 coupling flag so that lnd -> atm co2 fluxes are sent in all tests where this is possible, in order to test this coupling.
- Originally I had been thinking of a separate testmod to enable this and possibly other optional lnd -> atm couplings, but it seems better to just do this for all tests.
- However, this isn't crucial to do, since it would only cover a small number of lines beyond what's tested by adding FCO2 to the CTSM history file.
- If we do this, we can remove the addition of the FCO2 history field in the default test mod.
- Note that this might require having a separate "defaultbgc" testmod directory that inherits from default and that is included instead of default in at least some tests that include BGC; this "defaultbgc" testmod directory could add settings like this (and possibly some other history fields) that are unique to BGC.

- [ ] (2021-04-15) I think we could decrease our ERI testing: It's important to have some ERI testing, but it's probably sufficient to just have a few such tests. I would especially like to reduce or eliminate our ERI testing on izumi: with the multiple runs of this test, it is more prone to the periodic system failures we see on izumi, and there may be problems with rerunning ERI tests (as seen in https://github.com/ESCOMP/CTSM/pull/1324).