llnl / llnl/AMG2023

segmentation fault with >= 64 nodes on Frontier

Open
#13 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
9
Forks
18
PR merge metrics
No merged PRs in 30d

Description

I can run problem 1 successfully on Frontier with < 64 nodes fine, but I get a segmentation fault with >= 64 nodes:

Running with these driver parameters:
  Problem ID    = 1

=============================================
Hypre init times:
=============================================
Hypre init:
  wall clock time = 0.000006 seconds
  Laplacian_27pt:
    (Nx, Ny, Nz) = (1600, 1600, 1600)
    (Px, Py, Pz) = (8, 8, 8)

srun: error: frontier04522: tasks 282-287: Segmentation fault
srun: Terminating StepId=2131722.0

with Segmentation fault errors reported for all of the other MPI ranks as well.

I built Hypre v2.31.0 with:

./configure --with-hip --with-gpu-arch=gfx90a --with-MPI-lib-dirs="${MPICH_DIR}/lib" --with-MPI-libs="mpi" --with-MPI-include="${MPICH_DIR}/include" --enable-mixedint

with cce/17.0.0, rocm/5.7.1, and cray-mpich/8.1.28.

I'm running the problem with:

#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=7
#SBATCH --gpus-per-task=1
#SBATCH --gpu-bind=closest
#SBATCH -N 64

export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}
export MPICH_GPU_SUPPORT_ENABLED=1

srun ./amg -problem 1 -n 200 200 200 -P 8 8 8

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported srun ./amg -problem 1 -n 200 200 200 -P 8 8 8 run on Frontier with the stated Hypre, CCE, ROCm, and cray-mpich configuration. Trace the problem 1 execution and its behavior at fewer than 64 nodes versus 64 nodes or more. Done means the segmentation fault is explained and the run completes successfully at the reported scale.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.