Feature request: Testing CESM+DART infrastructure

Use case

It would be very helpful to people who use CESM+DART
if the released versions of CESM have been tested to ensure
that they provide the functionality required by DART,
such as the ability to build and run large, multi-instance jobs
and interact with DART as an External System Processing component.

Is your feature request related to a problem?

Recently developed CESM components can be used and tested with DART
only if the CESM infrastructure continues to enable it.
Often in the past CESM or CIME development has neglected to test for this use,
and the resulting release has been incompatible with DART (CESM issue #1807).
This required people who are not experts in CESM code to inefficiently find and suboptimally fix the problems.
This can take as long as it takes the CESM developers to release a new version,
resulting in a seemingly endless cycle, during which DART cannot be used with recent CESMs.

Describe your preferred solution

(Further) Integrate testing of the functionality required by DART into the CESM testing suite.
This is most important for major releases, which are most likely to be used with DART.
But it would be useful whenever a new version of a component model is made available
(if there's a DART interface to that component), so that DART can be used to evaluate it.
We anticipate that the CESM testing would not involve running an assimilation,
but just the features of CESM that enable that.
After passing the CESM tests, then people who want to do the assimilation
would do the full assimilation testing and model evaluation.

Describe any alternatives you have considered

See "Is this related to a problem"

The rest of the text is a discussion of strategies for implementing solutions.

Short list of issues

1. Current status of testing (@alperaltuntas)
2. Multi-instance + ensemble size
3. SourceMods and other modified code (@braczka , sea ice person?)
4. Compsets

Depth of testing

It's possible that a different level of testing could be done,
based on what parts of CESM had been upgraded (e.g. multi-instance scripting
vs a wave-model upgrade, which is currently irrelevant to DART)
or what level of release is being made (e.g. CESM2_3_1 vs CESM3).
? Does the testing currently vary, depending on these things?

Size of Ensemble

Several times in the past, tests using 2-3 members have passed,
but tests with larger numbers have failed, or shown unacceptable performance degradation.
My (Kevin's) intuition is that at least 10 members are required to see the latter.
For example, at one point the number of calls to the serial task, python build-namelist
was a function of the number of instances squared. That was not noticable
for 2 members, but for 80 it was 6400 calls.

Components

DART currently has interfaces to atm (CAM-FV, CAM-SE, WACCM(-X)), CLM5, POP(2?),
and CICE. The potential next interfaces are to a river or land ice model.
Work is underway to do assimilation with multiple components,
but that will (probably) be hidden within the ESP component and not require testing by CESM.
As of 2018-6 any compset that included a land ice component other than "stub"
couldn't be used, because all of those other land ice models cannot use the gregorian calendar.
A compset defined (2018) specifically for CAM6 assimilations is
FHIST_DARTC6 = HIST_CAM60_CLM50%SP_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV.

SourceMods

Each DART interface may currently require SourceMods in order to build
a model that works with DART effectively. These have been necessary partly because
of the lack of testing within CESM. It may be easy, or at least appropriate,
to include some of them into CESM, while others may not be. See attached files for examples.
There may be changes to CIME, which we implemented for DART,
which are not in the (cam.src) SourceMods. For example,
/glade/work/raeder/Models/cesm2_1_relsd_m5.6/cime/src/drivers/mct/cime_config/buildnml
has a time saving upgrade that just changes the log file name in modelio namelist files,
instead of regenerating them in every assimilation cycle.
There's no SourceMods mechanism for CIME code, that I know of, so it needs to be substituted manually.
? Do other components have nonSourceMods changes?

Size of the model(s)

We have not run into cases in which the resolution of the model was a factor
in the testing or functionality of the code. Of course, it's always possible
to exceed resources using a high resolution model, but that's not in the testing scope.
So testing a "large" ensemble may not require a large number of nodes, which can delay testing.