AMReX-Astro / AMReX-Astro/Castro
XRB with tracer particles fails with 2 levels of refinement (ref_ratio of 4 then 2)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 340
- Forks
- 105
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 8
Description
(on Cori Haswell nodes, using the 3e8 K setup)
The tracer particles implementation with the XRB flame wave simulations fails in the case of 2 refinement levels, ref_ratio of 4 then 2. It does not fail for 0, 1 refinement levels, and works for 2 levels if the ref_ratio is 2 and then 4, so the issue seems to be specifically in setting the first refinement factor (level 0 to level 1) to 4. It seems to be an error related to the AMReX ParticleContainer, with the backtrace indicating it's an issue in Redistribute after Castro regrids. The Backtrace shows:
3: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x20060f7b]
_ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii._omp_fn.2
../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:1270:194: /opt/gcc/8.3.0/snos/lib64/libgomp.so.1(GOMP_parallel+0x3f) [0x2aaaacccfb0f]
GOMP_parallel
../../../cray-gcc-8.3.0-201903122028.16ea96cb84a9a/libgomp/parallel.c:170:15: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x200679ae]
_ZNSt6vectorIPN5amrex12ParticleTileILi2ELi0ELi0ELi0ESaEESaIS3_EED4Ev inlined at ../../../external/amrex/Src/Base/AMReX_Vector.H:25:7 in _ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii
/opt/gcc/8.3.0/snos/include/g++/bits/stl_vector.h:567:15
_ZN5amrex6VectorIPNS_12ParticleTileILi2ELi0ELi0ELi0ESaEESaIS3_EED4Ev
../../../external/amrex/Src/Base/AMReX_Vector.H:25:7
_ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii
../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:1227:336: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x20049e62]
_ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE12RedistributeEiiii inlined at ../../../Source/driver/Castro.cpp:2217:31 in _ZN6Castro11post_regridEii
../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:929:5
_ZN6Castro11post_regridEii
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the 3e8 K XRB flame-wave case on Cori Haswell with two refinement levels and ref_ratio 4 then 2. Start at Castro.cpp:2217 in post_regrid and AMReX_ParticleContainerI.H around Redistribute lines 929 and 1270, comparing the working ref_ratio cases. Done means Redistribute no longer fails for the reported configuration while the listed working cases remain functional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100