QMCPACK / QMCPACK/qmcpack

NiO 512 atom runs hang using AMD rocSOLVER for matrix inversion

Open
#4,555 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 12h
Merged PRs (30d)
82

Description

**Describe the bug**
I noticed NiO run hangs when using rocsolver for matrix inversion.
The mystery is
a) 32 nodes with 8 MPI ranks on each node. Hang
b) 256 nodes with 1 MPI ranks on each node. Leave 56 cores and 7 GCDs idle. Runs fine.
The hang happens when using rocSolver regardless of using OpenMP offload or not.
So I believe the issue should at ROCr or below (like kernel drivers).

**To Reproduce**
reproducer input/output files.
This is a successful run.
[dmc-a512-e6144-gpu-Clang-batch-w7-run.zip](https://github.com/QMCPACK/qmcpack/files/11226710/dmc-a512-e6144-gpu-Clang-batch-w7-run.zip)
When a run fails, HIP stream synchronize never return.

Download hdf5 from https://github.com/QMCPACK/qmcpack/blob/9fe29deb747c86b8f9587ffe8e4a5eeee7fa27fe/tests/performance/NiO/README#L59
pseudopotential files can be found at
https://github.com/QMCPACK/qmcpack/tree/develop/tests/pseudopotentials_for_tests
Ni.opt.xml and O.ncpp.xml (need to be renamed to O.xml).

Froniter/Crusher build recipe, using rocm 5.4.3
https://github.com/QMCPACK/qmcpack/blob/develop/config/build_olcf_frontier_ROCm.sh
I was testing commit dbe06417bce423febef09c3fd2a36409c79be866 but 5d7a24f23b87151aaf56fab7e247cbf0e9398080 a later commit with the build recipe should also reproduce the issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.