QMCPACK / QMCPACK/qmcpack

NiO benchmark memory access fault on AMD GPU

Open
#4,553 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 12h
Merged PRs (30d)
82

Description

**Describe the bug**
Frontier/Crusher issue OLCFHELP-11849
```
[GPU Memory Error] Addr: 0x7fe5f5162000 Reason: No Idea!
Memory access fault by GPU node-11 (Agent handle: 0x13c9c80) on address 0x7fe5f5162000. Reason: Unknown.
```
failure is sporadic. Test may pass or fail from run to run. My current guess is that generated OpenMP offload kernels are buggy. Pass or fail are sensitive to timing.

I also tested running 1 thread per MPI rank and still observe failures.

**To Reproduce**
reproducer input/output files.
[memory-access-fault.zip](https://github.com/QMCPACK/qmcpack/files/11226455/memory-access-fault.zip)
[t1-failure.zip](https://github.com/QMCPACK/qmcpack/files/11226772/t1-failure.zip)

Download hdf5 from https://github.com/QMCPACK/qmcpack/blob/9fe29deb747c86b8f9587ffe8e4a5eeee7fa27fe/tests/performance/NiO/README#L59
pseudopotential files can be found at
https://github.com/QMCPACK/qmcpack/tree/develop/tests/pseudopotentials_for_tests
Ni.opt.xml and O.ncpp.xml (need to be renamed to O.xml).

Froniter/Crusher build recipe, using rocm 5.4.3
https://github.com/QMCPACK/qmcpack/blob/develop/config/build_olcf_frontier_ROCm.sh
I was testing commit dbe06417bce423febef09c3fd2a36409c79be866 but 5d7a24f23b87151aaf56fab7e247cbf0e9398080 a later commit with the build recipe should also reproduce the issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.