E3SM-Project / E3SM-Project/E3SM
Fail with DEBUG build running longer `ne30pg2_ne30pg2.F2010-SCREAMv1`
- Dominant language
- Fortran
- Stars
- 440
- Forks
- 481
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 36
Description
In this case, after 39 days, I get the following fail:
```
43: corrupted size vs. prev_size
43:
43: Program received signal SIGABRT: Process abort signal.
43:
43: Backtrace for this error:
43: #0 0x148582aac862 in ???
43: #1 0x148582aab8f5 in ???
43: #2 0x14858253cd6f in ???
43: #3 0x14858253ccdb in ???
43: #4 0x14858253e394 in ???
43: #5 0x148582582c37 in ???
43: #6 0x14858258acd9 in ???
43: #7 0x14858258b5a5 in ???
43: #8 0x14858258b722 in ???
43: #9 0x14858258dd3f in ???
43: #10 0x14858258f787 in ???
43: #11 0xe997ad in __canopyhydrologymod_MOD_canopyhydrology
43: at /dvs_ro/cfs/cdirs/e3sm/ndk/repos/c28-oct15/components/elm/src/biogeophys/CanopyHydrologyMod.F90:157
43: #12 0x6cf782 in __elm_driver_MOD_elm_drv._omp_fn.4
43: at /dvs_ro/cfs/cdirs/e3sm/ndk/repos/c28-oct15/components/elm/src/main/elm_driver.F90:1361
43: #13 0x14858fd30c4d in ???
43: #14 0x14858574f6e9 in ???
43: #15 0x14858260a53e in ???
```
This is standard ne30 case with a different START_DATE and perturbation than default setting.
What's odd here is that it fails after quite a bit of simulation.
In other DEBUG testing with similar setup, I've seen 4 different error messages (looks to all be in LND or ICE).
All cases are ok build in OPT.
I don't have a way of reproducing this with test string (as need to change start date and perturbation), but the case is here:
```
/pscratch/sd/n/ndk/e3sm_scratch/pm-gpu/c28-oct15/p.ne30pg2_ne30pg2.F2010-SCREAMv1.c28-oct15.1y.n0016.pert01.DEBUG
```
Note this is all with 128 vertical levels.
Contributor guide
Research direction
Start with the DEBUG case at /pscratch/sd/n/ndk/e3sm_scratch/pm-gpu/c28-oct15/p.ne30pg2_ne30pg2.F2010-SCREAMv1.c28-oct15.1y.n0016.pert01.DEBUG and inspect the failure locations in components/elm/src/biogeophys/CanopyHydrologyMod.F90:157 and components/elm/src/main/elm_driver.F90:1361. Compare DEBUG and OPT behavior if the case can be run; done requires identifying the cause of the delayed memory corruption and confirming a fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100