NiO benchmark memory access fault on AMD GPU
- Dominant language
- C++
- Stars
- 403
- Forks
- 154
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 82
Description
**Describe the bug**
Frontier/Crusher issue OLCFHELP-11849
```
[GPU Memory Error] Addr: 0x7fe5f5162000 Reason: No Idea!
Memory access fault by GPU node-11 (Agent handle: 0x13c9c80) on address 0x7fe5f5162000. Reason: Unknown.
```
failure is sporadic. Test may pass or fail from run to run. My current guess is that generated OpenMP offload kernels are buggy. Pass or fail are sensitive to timing.
I also tested running 1 thread per MPI rank and still observe failures.
**To Reproduce**
reproducer input/output files.
[memory-access-fault.zip](https://github.com/QMCPACK/qmcpack/files/11226455/memory-access-fault.zip)
[t1-failure.zip](https://github.com/QMCPACK/qmcpack/files/11226772/t1-failure.zip)
Download hdf5 from https://github.com/QMCPACK/qmcpack/blob/9fe29deb747c86b8f9587ffe8e4a5eeee7fa27fe/tests/performance/NiO/README#L59
pseudopotential files can be found at
https://github.com/QMCPACK/qmcpack/tree/develop/tests/pseudopotentials_for_tests
Ni.opt.xml and O.ncpp.xml (need to be renamed to O.xml).
Froniter/Crusher build recipe, using rocm 5.4.3
https://github.com/QMCPACK/qmcpack/blob/develop/config/build_olcf_frontier_ROCm.sh
I was testing commit dbe06417bce423febef09c3fd2a36409c79be866 but 5d7a24f23b87151aaf56fab7e247cbf0e9398080 a later commit with the build recipe should also reproduce the issue.
Contributor guide
Assessment
This issue has not been assessed yet.