QMCPACK / QMCPACK/qmcpack

Continuing a DMC run results in `nan` local energy.

Open
#4,487 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 12h
Merged PRs (30d)
82

Description

Continuing a DMC run results in `nan` local energy.

**Describe the bug**
I tried to continue a DMC run for this H2O example: https://github.com/QMCPACK/qmcpack/tree/develop/examples/molecules/H2O
and ran into the following error:
```
Fatal Error. Aborting at WalkerControlBase::sortWalkers
```
The `H2O.s003.dmc.dat` contains the following.
```
# Index LocalEnergy Variance Weight NumOfWalkers AvgSentWalkers TrialEnergy DiffEff LivingFraction
0 -1.7305745032e+01 2.7738748341e-01 -5.7724360458e+12 2.6880000000e+03 0.0000000000e+00 -1.7262185048e+01 9.9455568438e-01 0.0000000000e+00
1 nan nan nan 2.6880000000e+03 0.0000000000e+00 nan 9.9507990827e-01 1.0000000000e+00
```
All the files including inputs and outputs within the directory are uploaded here:
https://github.com/krongch2/h2o_qmc/tree/main/archive_cpu/2_dmc

**To Reproduce**
1. First, I edited the `simple-H2O.xml` file so that the field `checkpoint="-1"` becomes `checkpoint="0"` to ensure that the `config.h5` file is generated after the first DMC run.
2. Then, I ran the input file `simple-H2O.xml` by `bsub dmc.bsub.in`.
```
#!/bin/bash
#BSUB -P mat221
#BSUB -J dmc
#BSUB -o dmc.out
#BSUB -e dmc.err
#BSUB -W 00:20
#BSUB -nnodes 1
#BSUB -alloc_flags "smt1"

module load gcc/9.3.0 spectrum-mpi cuda essl netlib-lapack hdf5/1.10.7 fftw; module use /gpfs/alpine/mat151/world-shared/opt/modules; module load llvm/release-15.0.0-cuda11.0

NNODES=$(((LSB_DJOB_NUMPROC-1)/42))
RANKS_PER_NODE=6
RS_PER_NODE=6
exe_path=/gpfs/alpine/mat151/world-shared/opt/qmcpack/release-3.16.0/build_summit_Clang_offload_cuda_real/bin

export OMP_NUM_THREADS=7
jsrun -n $NNODES -a $RANKS_PER_NODE -c $((RANKS_PER_NODE*OMP_NUM_THREADS)) -g 6 -r 1 -d packed -b packed:$OMP_NUM_THREADS --smpiargs="-disable_gpu_hooks" $exe_path/qmcpack --enable-timers=fine simple-H2O.xml
```
The QMCPACK executable is located in `/gpfs/alpine/mat151/world-shared/opt/qmcpack/release-3.16.0/build_summit_Clang_offload_cuda_real/bin/qmcpack` on Summit.
This step autogenerates `H2O.s002.cont.xml` for the subsequent run.

3. I removed the VMC block from the `H2O.s002.cont.xml` file.

4. I ran the input file `H2O.s002.cont.xml` by submitting the following script with `bsub dmc.bsub.cont.in`
```
#!/bin/bash
#BSUB -P mat221
#BSUB -J dmc.cont
#BSUB -o dmc.cont.out
#BSUB -e dmc.cont.err
#BSUB -W 00:20
#BSUB -nnodes 1
#BSUB -alloc_flags "smt1"

module load gcc/9.3.0 spectrum-mpi cuda essl netlib-lapack hdf5/1.10.7 fftw; module use /gpfs/alpine/mat151/world-shared/opt/modules; module load llvm/release-15.0.0-cuda11.0

NNODES=$(((LSB_DJOB_NUMPROC-1)/42))
RANKS_PER_NODE=6
RS_PER_NODE=6
exe_path=/gpfs/alpine/mat151/world-shared/opt/qmcpack/release-3.16.0/build_summit_Clang_offload_cuda_real/bin

export OMP_NUM_THREADS=7
jsrun -n $NNODES -a $RANKS_PER_NODE -c $((RANKS_PER_NODE*OMP_NUM_THREADS)) -g 6 -r 1 -d packed -b packed:$OMP_NUM_THREADS --smpiargs="-disable_gpu_hooks" $exe_path/qmcpack --enable-timers=fine H2O.s002.cont.xml
```

**System:**
- Summit
- QMCPACK 3.16. Same issue for both `offload_cuda_real` and `Clang_cpu_cplx`
- module load gcc/9.3.0 spectrum-mpi cuda essl netlib-lapack hdf5/1.10.7 fftw; module use /gpfs/alpine/mat151/world-shared/opt/modules; module load llvm/release-15.0.0-cuda11.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.