ESCOMP / ESCOMP/CTSM

Threaded exact restart tests that change task/thread count fail with DGC or CISM fail when run to a year or more

Open
#4,160 10 comments 0 reactions 0 assignees View on GitHub
bug code health
Dominant language
Fortran
Stars
352
Forks
361
Avg merge
6d 6h
Merged PRs (30d)
8

Description

### Brief summary of bug

In bringing in #4110 I found that some of the longer threaded tests fail when using DGLC.

```
ERP_Ly3_P64x2.f10_f10_mg37.IHistClm50BgcCrop.derecho_intel.clm-cropMonthOutput ( COMPARE_base_rest )
ERP_Ly3_P64x2.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput--clm-matrixcnOn_ignore_warnings ( COMPARE_base_rest )
ERP_P64x2_Ld1096.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-clm50cropIrrigMonth_interp ( COMPARE_base_rest )
ERP_P64x2_Ld1096.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_o3falk_reduceOutput ( COMPARE_base_rest )
ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly ( COMPARE_base_rest )
ERP_P64x2_Ld396.f10_f10_mg37.IHistClm60Bgc.derecho_gnu.clm-monthly ( COMPARE_base_rest )
ERP_P64x2_Ld765.f10_f10_mg37.I2000Clm60BgcCrop.derecho_intel.clm-monthly ( COMPARE_base_rest )
```

It seems to be only tests that go beyond a year, at least that's my speculation.

### General bug information

**CTSM version you are using:** ctsm5.4.049-98-g48ed6abb9
**Does this bug cause significantly incorrect results in the model's science?** No
**Configurations affected:** ERP restart tests and PEM tests

### Details of bug

Looking at the threaded tests that fail comparison of rest to base. It's the cpl history files that fail for 4 fields:

``` shell
tail -n20 /glade/derecho/scratch/erik/tests_ctsm545acl/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int/run/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int.cpl.hi.2001-01-02-00000.nc.base.cprnc.out
grep RMS run/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int.cpl.hi.2001-01-02-00000.nc.base.cprnc.out
RMS rofImp_Forr_rofi_glc 7.0515E-21 NORMALIZED 8.4114E-14
RMS rofExp_Fgrg_rofi 7.3536E-22 NORMALIZED 9.1454E-15
RMS glc1Imp_Fgrg_rofi 5.0408E-21 NORMALIZED 1.3781E-15
RMS glc1Exp_Flgl_qice 4.7167E-21 NORMALIZED 4.4018E-16
```
so it's a roundoff difference somehow due to DGLC when threading is on.

### Important details of your setup / configuration so we can reproduce the bug

This was found in #4110 with this comment:

https://github.com/ESCOMP/CTSM/pull/4110#issuecomment-5200773303

### Important output or errors that show the problem

These are examples of tests that fail:

```
ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm60Sp.derecho_intel.clm-nofireemis ( COMPARE_base_rest )
PEA_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop ( RUN)
PEM_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop ( COMPARE_base_modpes )
```

and these are tests that PASS

```
ERI_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
ERS_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
PET_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
REP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
```

So it's not just exact restart that fails, or a fail when processors change, it's exact restart with a processor change. But, the PEM test fails while PET passes and those are with a change in processors. It's odd that PET passes (only changes OMP threads) while the PEM test fails (only changes MPI tasks), since it's only seen in threaded tests. I would think that would be switched.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the listed ERP, PEA, PEM, ERI, ERS, PET, and REP threaded tests, focusing on exact restarts with processor changes. Compare the cpl history cprnc outputs and the four reported fields in the 2001-01-02 output. Done means the affected restart comparisons pass without the reported roundoff mismatches.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.