bug: pert_wrf_bc code introduces artifacts to perturbed WRF boundary files

Describe the bug

Potential bug reported by Rui Cheng (UCSD). See dart(at)ucar.edu for more details.

1. Recreate bug:
   Using the pert_wrf_bc code when running a WRF-DART simulation. The pert_wrf_bc code is invoked both when running init_ensemble_var.csh and driver.csh within the WRF shell scripting.

2. What was the expected outcome?
   The perturbation of the wrfbdy file should reflect the WRFDA CV3 perturbation which is applied to the wrfinput file, then conveyed to the
   wrfbdy files. Perturbations are applied to atmospheric variables (e.g. wind, geopotential, temp, precip fields) to enhance ensemble spread
   during DA.

3. What actually happened? ### Error Message Please provide any error messages.

Artifacts are introduced within the PH (perturbed geopotential) field, that are unrelated to the expected perturbations introduced through perturbation bank. There is no specific error message -- because there is no quality checks performed on the perturbed fields -- however, this could lead to instabilities during subsequent WRF integration (e.g. CFL errors etc).

These artifacts seems to coincide with mountainous terrain. So if the boundary conditions do not coincide with mountainous terrain, this may not be noticeable.

Screenshots

Below I show a screenshot of the wrfbdy fields for U, V, W, PH, T, and QVAPOR for each of the wrf domain boundaries within the WRF-DART tutorial. The wrfbdy 'mean' files generated from the gen_retro_icbc.csh step should be similar (nearly identical) to the mean of the perturbed ensemble of wrfbdy files generated by pert_wrf_bc. If there are signficant pattern differences -- this indicates a bug. Note the artifact for the PH variable (perturbed geopotential) for the southern boundary (BYS_pertmean). Red arrow. The fact that there is a difference between BYS_mean and BYS_pertmean indicates a problem.

Definitions:

BYS_mean: southern boundary generated from gen_retro_icbc_csh
BYS_pertmean: mean of perturbed southern boundary ensemble members (pert_wrf_bc)

BYE_mean: northern boundary generated from gen_retro_icbc_csh
BYE_pertmean: mean of perturbed northern boundary ensemble members (pert_wrf_bc)

BXS_mean: western boundary generated from gen_retro_icbc_csh
BXS_pertmean: mean of perturbed western boundary ensemble members (pert_wrf_bc)

BXE_mean: eastern boundary generated from gen_retro_icbc_csh
BXE_pertmean: mean of perturbed eastern boundary ensemble members (pert_wrf_bc)

Version of DART

Latest release.

Have you modified the DART code?

No, but a proposed fix includes replacing the existing lines within pert_wrf_bc with something more related to current WRFv4+ code..

This fix is suggested to work by Rui. I am currently in contact with MMM to verify this fix as well.

Build information

Derecho build with gfortran