E3SM-Project / E3SM-Project/E3SM
Seg-fault in scream_init_atm for ne4 at 96 MPI tasks
- Dominant language
- Fortran
- Stars
- 441
- Forks
- 481
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 33
Description
These cases are seg-faulting in the 2nd/restart run on 8 nodes at 12 ppn:
```
1. ERS.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.aurora_oneapi-ifxgpu
2. ERS.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.aurora_oneapi-ifxgpu.eamxx-mam4xx-aci
3. ERS.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.aurora_oneapi-ifxgpu.eamxx-mam4xx-aero_microphysics
4. ERS.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.aurora_oneapi-ifxgpu.eamxx-mam4xx-drydep
5. ERS.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.aurora_oneapi-ifxgpu.eamxx-mam4xx-optics
6. ERS.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.aurora_oneapi-ifxgpu.eamxx-mam4xx-wetscav
7. ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.aurora_oneapi-ifxgpu.eamxx-small_kernels--eamxx-output-preset-5
8. ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.aurora_oneapi-ifxgpu.eamxx-small_kernels_p3--eamxx-output-preset-5
9. ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.aurora_oneapi-ifxgpu.eamxx-small_kernels_shoc--eamxx-output-preset-5
10. ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.aurora_oneapi-ifxgpu.eamxx-output-preset-2
```
https://my.cdash.org/viewTest.php?onlydelta&buildid=2962487
Looked at the first case and it runs fine on 1 node (12 MPI) and 2 nodes (24 MPI).
Backtrace:
```
#0 0x000000000219f8ef in operator= () at /lus/flare/projects/E3SM_Dec/azamat/scratch/ERS.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.aurora_oneapi-ifxgpu.JNextGpucxx20250618_061600/bld/oneapi-ifxgpu/mpich/nodebug/nothreads/include/View/Kokkos_ViewLegacy.hpp:907
#1 finalize () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/../eam/src/physics/rrtmgp/external/cpp/rrtmgp/mo_gas_optics_rrtmgp.h:1345
#2 0x0000000002022ba9 in rrtmgp_finalize () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/physics/rrtmgp/eamxx_rrtmgp_interface.hpp:482
#3 finalize_impl () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/physics/rrtmgp/eamxx_rrtmgp_process_interface.cpp:1242
#4 0x000000000258660b in finalize_impl () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/share/atm_process/atmosphere_process_group.cpp:485
#5 0x000000000258660b in finalize_impl () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/share/atm_process/atmosphere_process_group.cpp:485
#6 0x00000000013d70c1 in finalize () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/control/atmosphere_driver.cpp:1710
#7 0x00000000013d6671 in ~AtmosphereDriver () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/control/atmosphere_driver.cpp:105
#8 0x000000000058463b in _M_release () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/shared_ptr_base.h:346
#9 ~__shared_count () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/shared_ptr_base.h:1071
#10 ~__shared_ptr () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/shared_ptr_base.h:1524
#11 ~holder () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/externals/ekat/src/ekat/std_meta/ekat_std_any.hpp:40
#12 ~holder () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/externals/ekat/src/ekat/std_meta/ekat_std_any.hpp:40
#13 0x0000000000586873 in _M_release () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/shared_ptr_base.h:346
#14 ~__shared_count () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/shared_ptr_base.h:1071
#15 ~__shared_ptr () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/shared_ptr_base.h:1524
#16 ~any () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/externals/ekat/src/ekat/std_meta/ekat_std_any.hpp:25
#17 ~pair () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/stl_pair.h:187
#18 destroy > () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/new_allocator.h:198
#19 destroy > () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/alloc_traits.h:558
#20 _M_destroy_node () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/stl_tree.h:625
#21 _M_drop_node () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/stl_tree.h:633
#22 _M_erase () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/stl_tree.h:1938
#23 0x00000000005868a2 in _M_erase () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/stl_tree.h:1936
#24 0x000000000057556d in clear () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/stl_tree.h:1255
#25 clear () at /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/gcc-13.3.0-4enwbrb/lib/gcc/x86_64-pc-linux-gnu/13.3.0/../../../../include/c++/13.3.0/bits/stl_map.h:1184
#26 clean_up () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/mct_coupling/ScreamContext.hpp:52
#27 0x000000000056c3ce in fpe_guard_wrapper<(lambda at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/mct_coupling/eamxx_cxx_f90_interface.cpp:255:21)> () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/mct_coupling/eamxx_cxx_f90_interface.cpp:66
#28 scream_init_atm () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/mct_coupling/eamxx_cxx_f90_interface.cpp:255
#29 0x000000000056843e in atm_comp_mct::atm_init_mct (eclock=..., cdata=..., x2a=..., a2x=..., nlfilename=..., _nlfilename=) at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/components/eamxx/src/mct_coupling/atm_comp_mct.F90:282
#30 0x000000000046b367 in component_mod::component_init_cc (eclock=..., comp=..., comp_init=0x56415741e5894855, infodata=..., nlfilename=..., seq_flds_x2c_fluxes=, seq_flds_c2x_fluxes=..., _nlfilename=6, _seq_flds_x2c_fluxes=0, _seq_flds_c2x_fluxes=0) at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/driver-mct/main/component_mod.F90:257
#31 0x000000000044ab95 in cime_comp_mod::cime_init () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/driver-mct/main/cime_comp_mod.F90:1502
#32 0x000000000046a2cd in cime_driver () at /lus/flare/projects/E3SM_Dec/gitlabci/E3SM/driver-mct/main/cime_driver.F90:122
```
Contributor guide
Research direction
Reproduce the listed restart cases at 12, 24, and 96 MPI tasks, then inspect scream_init_atm in components/eamxx/src/mct_coupling/eamxx_cxx_f90_interface.cpp and the finalize path in mo_gas_optics_rrtmgp.h. Use the supplied backtrace and CDash run as starting points. Done means the affected 96-task restart runs no longer seg-fault.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, fortran, hpc
- Domain
- distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100