E3SM-Project / E3SM-Project/E3SM

Fail with DEBUG build running longer `ne30pg2_ne30pg2.F2010-SCREAMv1`

Open
#7,802 6 comments 1 reaction 0 assignees View on GitHub
ELM MPAS-seaice
Dominant language
Fortran
Stars
440
Forks
481
Avg merge
4d 6h
Merged PRs (30d)
36

Description

In this case, after 39 days, I get the following fail:

```
43: corrupted size vs. prev_size
43:
43: Program received signal SIGABRT: Process abort signal.
43:
43: Backtrace for this error:
43: #0 0x148582aac862 in ???
43: #1 0x148582aab8f5 in ???
43: #2 0x14858253cd6f in ???
43: #3 0x14858253ccdb in ???
43: #4 0x14858253e394 in ???
43: #5 0x148582582c37 in ???
43: #6 0x14858258acd9 in ???
43: #7 0x14858258b5a5 in ???
43: #8 0x14858258b722 in ???
43: #9 0x14858258dd3f in ???
43: #10 0x14858258f787 in ???
43: #11 0xe997ad in __canopyhydrologymod_MOD_canopyhydrology
43: at /dvs_ro/cfs/cdirs/e3sm/ndk/repos/c28-oct15/components/elm/src/biogeophys/CanopyHydrologyMod.F90:157
43: #12 0x6cf782 in __elm_driver_MOD_elm_drv._omp_fn.4
43: at /dvs_ro/cfs/cdirs/e3sm/ndk/repos/c28-oct15/components/elm/src/main/elm_driver.F90:1361
43: #13 0x14858fd30c4d in ???
43: #14 0x14858574f6e9 in ???
43: #15 0x14858260a53e in ???
```

This is standard ne30 case with a different START_DATE and perturbation than default setting.
What's odd here is that it fails after quite a bit of simulation.
In other DEBUG testing with similar setup, I've seen 4 different error messages (looks to all be in LND or ICE).
All cases are ok build in OPT.

I don't have a way of reproducing this with test string (as need to change start date and perturbation), but the case is here:
```
/pscratch/sd/n/ndk/e3sm_scratch/pm-gpu/c28-oct15/p.ne30pg2_ne30pg2.F2010-SCREAMv1.c28-oct15.1y.n0016.pert01.DEBUG
```

Note this is all with 128 vertical levels.

Contributor guide

Open the contributing guide

Research direction

Start with the DEBUG case at /pscratch/sd/n/ndk/e3sm_scratch/pm-gpu/c28-oct15/p.ne30pg2_ne30pg2.F2010-SCREAMv1.c28-oct15.1y.n0016.pert01.DEBUG and inspect the failure locations in components/elm/src/biogeophys/CanopyHydrologyMod.F90:157 and components/elm/src/main/elm_driver.F90:1361. Compare DEBUG and OPT behavior if the case can be run; done requires identifying the cause of the delayed memory corruption and confirming a fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
hpc
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.