ESCOMP / ESCOMP/CTSM

Threaded tests are slow on Derecho

Open
#2,289 7 comments 0 reactions 1 assignee Claimed by @ekluzek View on GitHub
bug
Dominant language
Fortran
Stars
352
Forks
361
Avg merge
2d 21h
Merged PRs (30d)
7

Description

### Brief summary of bug

Longer threaded tests seem to be hanging on Derecho with release-cesm2.2.04

ERP_D_Ld10_P64x2.f10_f10_musgs.IHistClm50BgcCrop.derecho_intel.clm-ciso_decStart
ERP_Ly3_P64x2.f10_f10_musgs.IHistClm50BgcCrop.derecho_intel.clm-cropMonthOutput
ERP_P64x2_Lm25.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-monthly
ERP_P64x2_Lm36.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-clm50cropIrrigMonth_interp
ERP_P64x2_Lm7.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly
ERP_P64x2_Ly3.f10_f10_musgs.I2000Clm50BgcCrop.derecho_intel.clm-irrig_o3_reduceOutput
ERS_Ly3_P64x2.f10_f10_musgs.IHistClm50BgcCropG.derecho_intel.clm-cropMonthOutput

### General bug information

**CTSM version you are using:** release-cesm2.2.03-12-g1b4fa3602

**Does this bug cause significantly incorrect results in the model's science?** No?

**Configurations affected:** Threaded tests that go more than a year and a half simulation years

### Details of bug

These tests were originally on Cheyenne with a PE layout of 36x2 which worked in release-cesm2.2.02. On Derecho they seem to hang after it gets longer and run out of wallclock time of 1:40.

### Important output or errors that show the problem

cesm.log:
``` text
dec0275.hsn.de.hpc.ucar.edu 63: forrtl: error (78): process killed (SIGTERM)
dec0275.hsn.de.hpc.ucar.edu 63: Image PC Routine Line Source
dec0275.hsn.de.hpc.ucar.edu 63: libpthread-2.31.s 0000145589C668C0 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455890100C5 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455879B60C9 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455879EE0D6 Unknown Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpi_intel.so.1 00001455879EF31F MPI_Wait Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: libmpifort_intel. 00001455897EA3BC mpi_wait Unknown Unknown
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 000000000131B9C8 m_spmdutils_mp_m_ 877 m_SPMDutils.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 000000000130F1E2 m_rearranger_mp_r 1132 m_Rearranger.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000001301CA1 m_matattrvectmul_ 514 m_MatAttrVectMul.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000CD46A1 mosart_physics_mo 145 MOSART_physics_mod.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000CA4D53 rtmmod_mp_rtmrun_ 1830 RtmMod.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000C883AC rof_comp_mct_mp_r 299 rof_comp_mct.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 0000000000436B2E component_mod_mp_ 737 component_mod.F90
dec0275.hsn.de.hpc.ucar.edu 63: cesm.exe 000000000041BFB0 cime_comp_mod_mp_ 2638 cime_comp_mod.F90
```

lnd.log:
``` text
nstep = 23926 TS = 274.112624074611290
nstep = 23927 TS = 274.085864720367340
nstep = 23928 TS = 274.022571240220486
nstep = 23929 TS = 273.960032305360414
nstep = 23930 TS = 274.039398595550153
nstep = 23931 TS = 273.989408017349092
(shr_dmodel_readstrm) file ub: /glade/campaign/cesm/cesmdata/inputdata/atm/datm7/NASA_LIS/clmforc.Li_2012_climo1995-2011.T62.lnfm_Total_c140423.nc 1070
nstep = 23932 TS = 273.916223972565433
nstep = 23933 TS = 273.782815379037402
nstep = 23934 TS = 273.618251977645627
nstep = 23935 TS = 273.452100094584694
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.