AMReX-Astro / AMReX-Astro/Castro

XRB with tracer particles fails with 2 levels of refinement (ref_ratio of 4 then 2)

Open
#1,616 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
340
Forks
105
Avg merge
3d 8h
Merged PRs (30d)
8

Description

(on Cori Haswell nodes, using the 3e8 K setup)
The tracer particles implementation with the XRB flame wave simulations fails in the case of 2 refinement levels, ref_ratio of 4 then 2. It does not fail for 0, 1 refinement levels, and works for 2 levels if the ref_ratio is 2 and then 4, so the issue seems to be specifically in setting the first refinement factor (level 0 to level 1) to 4. It seems to be an error related to the AMReX ParticleContainer, with the backtrace indicating it's an issue in Redistribute after Castro regrids. The Backtrace shows:

3: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x20060f7b]
_ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii._omp_fn.2
../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:1270:19

4: /opt/gcc/8.3.0/snos/lib64/libgomp.so.1(GOMP_parallel+0x3f) [0x2aaaacccfb0f]
GOMP_parallel
../../../cray-gcc-8.3.0-201903122028.16ea96cb84a9a/libgomp/parallel.c:170:1

5: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x200679ae]
_ZNSt6vectorIPN5amrex12ParticleTileILi2ELi0ELi0ELi0ESaEESaIS3_EED4Ev inlined at ../../../external/amrex/Src/Base/AMReX_Vector.H:25:7 in _ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii
/opt/gcc/8.3.0/snos/include/g++/bits/stl_vector.h:567:15
_ZN5amrex6VectorIPNS_12ParticleTileILi2ELi0ELi0ELi0ESaEESaIS3_EED4Ev
../../../external/amrex/Src/Base/AMReX_Vector.H:25:7
_ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii
../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:1227:33

6: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x20049e62]
_ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE12RedistributeEiiii inlined at ../../../Source/driver/Castro.cpp:2217:31 in _ZN6Castro11post_regridEii
../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:929:5
_ZN6Castro11post_regridEii

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the 3e8 K XRB flame-wave case on Cori Haswell with two refinement levels and ref_ratio 4 then 2. Start at Castro.cpp:2217 in post_regrid and AMReX_ParticleContainerI.H around Redistribute lines 929 and 1270, comparing the working ref_ratio cases. Done means Redistribute no longer fails for the reported configuration while the listed working cases remain functional.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.