segmentation fault with >= 64 nodes on Frontier
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 9
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
I can run problem 1 successfully on Frontier with < 64 nodes fine, but I get a segmentation fault with >= 64 nodes:
Running with these driver parameters:
Problem ID = 1
=============================================
Hypre init times:
=============================================
Hypre init:
wall clock time = 0.000006 seconds
Laplacian_27pt:
(Nx, Ny, Nz) = (1600, 1600, 1600)
(Px, Py, Pz) = (8, 8, 8)
srun: error: frontier04522: tasks 282-287: Segmentation fault
srun: Terminating StepId=2131722.0
with Segmentation fault errors reported for all of the other MPI ranks as well.
I built Hypre v2.31.0 with:
./configure --with-hip --with-gpu-arch=gfx90a --with-MPI-lib-dirs="${MPICH_DIR}/lib" --with-MPI-libs="mpi" --with-MPI-include="${MPICH_DIR}/include" --enable-mixedint
with cce/17.0.0, rocm/5.7.1, and cray-mpich/8.1.28.
I'm running the problem with:
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=7
#SBATCH --gpus-per-task=1
#SBATCH --gpu-bind=closest
#SBATCH -N 64
export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}
export MPICH_GPU_SUPPORT_ENABLED=1
srun ./amg -problem 1 -n 200 200 200 -P 8 8 8
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported srun ./amg -problem 1 -n 200 200 200 -P 8 8 8 run on Frontier with the stated Hypre, CCE, ROCm, and cray-mpich configuration. Trace the problem 1 execution and its behavior at fewer than 64 nodes versus 64 nodes or more. Done means the segmentation fault is explained and the run completes successfully at the reported scale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100