E3SM-Project / E3SM-Project/E3SM

SIGTERM error when running ultra-low resolution on Cori

Open
#3,380 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Fortran
Stars
440
Forks
481
Avg merge
4d 7h
Merged PRs (30d)
33

Description

I am trying to run the E3SM at an ultra-low resolution on cori but keep getting the following error:

> 631: forrtl: error (78): process killed (SIGTERM)
631: Image PC Routine Line Source
631: e3sm.exe 0000000004428CEE Unknown Unknown Unknown
631: e3sm.exe 0000000003C9A300 Unknown Unknown Unknown
631: e3sm.exe 0000000004388761 Unknown Unknown Unknown
631: e3sm.exe 0000000004382948 Unknown Unknown Unknown
631: e3sm.exe 0000000003E7FF3F Unknown Unknown Unknown
631: e3sm.exe 0000000003E8099A Unknown Unknown Unknown
631: e3sm.exe 0000000003E68CE6 Unknown Unknown Unknown
631: e3sm.exe 0000000003E3057E Unknown Unknown Unknown
631: e3sm.exe 0000000003E30807 Unknown Unknown Unknown
631: e3sm.exe 0000000003F22E8C Unknown Unknown Unknown
631: e3sm.exe 0000000003F24D0F Unknown Unknown Unknown
631: e3sm.exe 0000000003E1C66B Unknown Unknown Unknown
631: e3sm.exe 0000000003E1CEBA Unknown Unknown Unknown
631: e3sm.exe 0000000003E2E68A Unknown Unknown Unknown
631: e3sm.exe 00000000026E79CA shr_mpi_mod_mp_sh 676 shr_mpi_mod.F90
631: e3sm.exe 00000000025A33DA seq_infodata_mod_ 2522 seq_infodata_mod.F90
631: e3sm.exe 000000000042500B component_mod_mp_ 906 component_mod.F90
631: e3sm.exe 000000000040C0AA cime_comp_mod_mp_ 3629 cime_comp_mod.F90
631: e3sm.exe 00000000004041AD cime_comp_mod_mp_ 2762 cime_comp_mod.F90
631: e3sm.exe 000000000042406D MAIN__ 133 cime_driver.F90
631: e3sm.exe 0000000000401ADE Unknown Unknown Unknown
631: e3sm.exe 000000000453BE4F Unknown Unknown Unknown
631: e3sm.exe 00000000004019CA Unknown Unknown Unknown
srun: error: nid03374: tasks 202-267: Exited with exit code 1
srun: error: nid03378: tasks 470-535: Exited with exit code 1
srun: error: nid03382: tasks 738-803: Exited with exit code 1
srun: error: nid03375: tasks 269-334: Exited with exit code 1
srun: error: nid03381: tasks 671-736: Exited with exit code 1
srun: error: nid03376: tasks 336-401: Exited with exit code 1
srun: error: nid03372: tasks 84,95-96,98,114,117,121,128-133: Exited with exit code 1
srun: error: nid03373: tasks 135-200: Exited with exit code 1
srun: error: nid03371: tasks 29-30,43,59: Exited with exit code 1
srun: error: nid03380: tasks 604-669: Exited with exit code 1

I have edited the radlw.f90 for a project I am working on so I am unsure if this error is because of that or something else. Any help debugging this error would be much appreciated!

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the ultra-low-resolution run on Cori and reviewing the changes in radlw.f90. Then inspect the reported Fortran stack frames, including shr_mpi_mod.F90, seq_infodata_mod.F90, component_mod.F90, cime_comp_mod.F90, and cime_driver.F90. Done means identifying whether the SIGTERM comes from the edit or the run configuration and documenting a reproducible resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.