QMCPACK / QMCPACK/qmcpack

NaN Values in Gradients Cause Calculation Abortion Using Mixed Precision with GPU Offload

Open
#5,015 3 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 12h
Merged PRs (30d)
82

Description

**Describe the bug**
When running a periodic calculation with 9 twists, 724 electrons, and 44 atoms using the mixed precision version of QMCPACK with GPU offload, the calculation was aborted with the following error:
```
NaNguard::checkOneParticleGradients error message: TWF::calcRatioGrad at particle 687
grads[0] = (-nan,0.0418255)
grads[1] = (-nan,-0.0806002)
grads[2] = (-nan,0.0412396)
Unexpected exception thrown in threaded section
Fatal Error. Aborting at Unhandled Exception
This issue appears to be related to NaN values in the gradients of the wave function for a specific particle.
```
The same calculation with full precision ran smoothly without any problems.

**To Reproduce**
Input and output files below:
[dmc_2x2_single_prec-test.zip](https://github.com/user-attachments/files/15506869/dmc_2x2_single_prec-test.zip)

**Expected behavior**
The calculation should complete successfully without encountering NaN values in the wave function gradients, resulting in accurate and stable output data.

**System:**
System name: Perlmutter
Modules loaded:
module use /global/common/software/nersc/n9/llvm/modules
module load craype cray-mpich
module load llvm/17.0.6-gpu
Other systems where this is reproducible: Not tested on other systems.

**Additional context**
The calculation was performed using the complex version of QMCPACK with NVIDIA GPU and OpenMP offload.
No other context or error messages where in the output files.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.