Threaded tests are slow on Derecho
- Dominant language
- Fortran
- Stars
- 352
- Forks
- 361
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 7
Description
### Brief summary of bug
Longer threaded tests seem to be hanging on Derecho with release-cesm2.2.04
ERP_D_Ld10_P64x2.f10_f10_musgs.IHistClm50BgcCrop.derecho_intel.clm-ciso_decStart
ERP_Ly3_P64x2.f10_f10_musgs.IHistClm50BgcCrop.derecho_intel.clm-cropMonthOutput
ERP_P64x2_Lm25.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-monthly
ERP_P64x2_Lm36.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-clm50cropIrrigMonth_interp
ERP_P64x2_Lm7.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly
ERP_P64x2_Ly3.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-irrig_o3_reduceOutput
ERS_Ly3_P64x2.f10_f10_musgs.IHistClm50BgcCropG.derecho_intel.clm-cropMonthOutput
### General bug information
**CTSM version you are using:** release-cesm2.2.03-12-g1b4fa3602
**Does this bug cause significantly incorrect results in the model's science?** No?
**Configurations affected:** Threaded tests that go more than a year and a half simulation years
### Details of bug
These tests were originally on Cheyenne with a PE layout of 36x2 which worked in release-cesm2.2.02. On Derecho they seem to hang after it gets longer and run out of wallclock time of 1:40.
### Important output or errors that show the problem
cesm.log:
``` text
dec0275.hsn.de.hpc.ucar.edu 63: forrtl: error (78): process killed (SIGTERM)
dec0275.hsn.de.hpc.ucar.edu 63: Image PC Routine Line Source
dec0275.hsn.de.hpc.ucar.edu 63: libpthread-2.31.s 0000145589C668C0 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455890100C5 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455879B60C9 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455879EE0D6 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455879EF31F MPI_Wait Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpifort_intel. 00001455897EA3BC mpi_wait Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 000000000131B9C8 m_spmdutils_mp_m_ 877 m_SPMDutils.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 000000000130F1E2 m_rearranger_mp_r 1132 m_Rearranger.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000001301CA1 m_matattrvectmul_ 514 m_MatAttrVectMul.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000CD46A1 mosart_physics_mo 145 MOSART_physics_mod.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000CA4D53 rtmmod_mp_rtmrun_ 1830 RtmMod.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000C883AC rof_comp_mct_mp_r 299 rof_comp_mct.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000436B2E component_mod_mp_ 737 component_mod.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 000000000041BFB0 cime_comp_mod_mp_ 2638 cime_comp_mod.F90
```
lnd.log:
``` text
nstep = 23926 TS = 274.112624074611290
nstep = 23927 TS = 274.085864720367340
nstep = 23928 TS = 274.022571240220486
nstep = 23929 TS = 273.960032305360414
nstep = 23930 TS = 274.039398595550153
nstep = 23931 TS = 273.989408017349092
(shr_dmodel_readstrm) file ub: /glade/campaign/cesm/cesmdata/inputdata/atm/datm7/NASA_LIS/clmforc.Li_2012_climo1995-2011.T62.lnfm_Total_c140423.nc 1070
nstep = 23932 TS = 273.916223972565433
nstep = 23933 TS = 273.782815379037402
nstep = 23934 TS = 273.618251977645627
nstep = 23935 TS = 273.452100094584694
```
Contributor guide
Assessment
This issue has not been assessed yet.