NiO 512 atom runs hang using AMD rocSOLVER for matrix inversion
- Dominant language
- C++
- Stars
- 403
- Forks
- 154
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 82
Description
**Describe the bug**
I noticed NiO run hangs when using rocsolver for matrix inversion.
The mystery is
a) 32 nodes with 8 MPI ranks on each node. Hang
b) 256 nodes with 1 MPI ranks on each node. Leave 56 cores and 7 GCDs idle. Runs fine.
The hang happens when using rocSolver regardless of using OpenMP offload or not.
So I believe the issue should at ROCr or below (like kernel drivers).
**To Reproduce**
reproducer input/output files.
This is a successful run.
[dmc-a512-e6144-gpu-Clang-batch-w7-run.zip](https://github.com/QMCPACK/qmcpack/files/11226710/dmc-a512-e6144-gpu-Clang-batch-w7-run.zip)
When a run fails, HIP stream synchronize never return.
Download hdf5 from https://github.com/QMCPACK/qmcpack/blob/9fe29deb747c86b8f9587ffe8e4a5eeee7fa27fe/tests/performance/NiO/README#L59
pseudopotential files can be found at
https://github.com/QMCPACK/qmcpack/tree/develop/tests/pseudopotentials_for_tests
Ni.opt.xml and O.ncpp.xml (need to be renamed to O.xml).
Froniter/Crusher build recipe, using rocm 5.4.3
https://github.com/QMCPACK/qmcpack/blob/develop/config/build_olcf_frontier_ROCm.sh
I was testing commit dbe06417bce423febef09c3fd2a36409c79be866 but 5d7a24f23b87151aaf56fab7e247cbf0e9398080 a later commit with the build recipe should also reproduce the issue.
Contributor guide
Assessment
This issue has not been assessed yet.