Threaded exact restart tests that change task/thread count fail with DGC or CISM fail when run to a year or more
- Dominant language
- Fortran
- Stars
- 352
- Forks
- 361
- Avg merge
- 6d 6h
- Merged PRs (30d)
- 8
Description
### Brief summary of bug
In bringing in #4110 I found that some of the longer threaded tests fail when using DGLC.
```
ERP_Ly3_P64x2.f10_f10_mg37.IHistClm50BgcCrop.derecho_intel.clm-cropMonthOutput ( COMPARE_base_rest )
ERP_Ly3_P64x2.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput--clm-matrixcnOn_ignore_warnings ( COMPARE_base_rest )
ERP_P64x2_Ld1096.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-clm50cropIrrigMonth_interp ( COMPARE_base_rest )
ERP_P64x2_Ld1096.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_o3falk_reduceOutput ( COMPARE_base_rest )
ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly ( COMPARE_base_rest )
ERP_P64x2_Ld396.f10_f10_mg37.IHistClm60Bgc.derecho_gnu.clm-monthly ( COMPARE_base_rest )
ERP_P64x2_Ld765.f10_f10_mg37.I2000Clm60BgcCrop.derecho_intel.clm-monthly ( COMPARE_base_rest )
```
It seems to be only tests that go beyond a year, at least that's my speculation.
### General bug information
**CTSM version you are using:** ctsm5.4.049-98-g48ed6abb9
**Does this bug cause significantly incorrect results in the model's science?** No
**Configurations affected:** ERP restart tests and PEM tests
### Details of bug
Looking at the threaded tests that fail comparison of rest to base. It's the cpl history files that fail for 4 fields:
``` shell
tail -n20 /glade/derecho/scratch/erik/tests_ctsm545acl/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int/run/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int.cpl.hi.2001-01-02-00000.nc.base.cprnc.out
grep RMS run/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int.cpl.hi.2001-01-02-00000.nc.base.cprnc.out
RMS rofImp_Forr_rofi_glc 7.0515E-21 NORMALIZED 8.4114E-14
RMS rofExp_Fgrg_rofi 7.3536E-22 NORMALIZED 9.1454E-15
RMS glc1Imp_Fgrg_rofi 5.0408E-21 NORMALIZED 1.3781E-15
RMS glc1Exp_Flgl_qice 4.7167E-21 NORMALIZED 4.4018E-16
```
so it's a roundoff difference somehow due to DGLC when threading is on.
### Important details of your setup / configuration so we can reproduce the bug
This was found in #4110 with this comment:
https://github.com/ESCOMP/CTSM/pull/4110#issuecomment-5200773303
### Important output or errors that show the problem
These are examples of tests that fail:
```
ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm60Sp.derecho_intel.clm-nofireemis ( COMPARE_base_rest )
PEA_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop ( RUN)
PEM_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop ( COMPARE_base_modpes )
```
and these are tests that PASS
```
ERI_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
ERS_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
PET_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
REP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
```
So it's not just exact restart that fails, or a fail when processors change, it's exact restart with a processor change. But, the PEM test fails while PET passes and those are with a change in processors. It's odd that PET passes (only changes OMP threads) while the PEM test fails (only changes MPI tasks), since it's only seen in threaded tests. I would think that would be switched.
Contributor guide
Research direction
Start by reproducing the listed ERP, PEA, PEM, ERI, ERS, PET, and REP threaded tests, focusing on exact restarts with processor changes. Compare the cpl history cprnc outputs and the four reported fields in the 2001-01-02 output. Done means the affected restart comparisons pass without the reported roundoff mismatches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100