QMCPACK / QMCPACK/qmcpack

Treatment of random number seeds in ensemble/batched runs

Open
#4,095 6 comments 0 reactions 0 assignees View on GitHub
bug discussion enhancement
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 13h
Merged PRs (30d)
90

Description

The purpose of this issue is to discuss if the following behavior is to be considered as a bug or a feature, and then what to do/not do about it.

While setting up minimal tests for the ensemble / batched run functionality (#4091 , #4093), I noticed that the treatment of random number seeds is different in ensemble runs. The merged PRs currently check only for a crash and the statistical results are not yet verified.

The key difference is that in an ensemble run the seeding &/or random number use is different so that with a fixed seed and the same inputs, every run in the ensemble will do a distinct QMC run. This applies even to the first input which will give different results from when run independently. The results also depend on the size of the ensemble.

This historical choice has the consequence that none of the deterministic tests can be used to check ensemble runs, and more generally that someone using fixed seeds for reproducibility and only using ensembles for HPC throughput reasons will not get the results that they expect.

Hopefully no one has been caught out by this. Clearly the behavior needs to be documented. The question is then whether we should change the behavior and what behavior(s) would best suit different workflows.

The following illustrates the behavior with different ensemble sizes. If the seeds were treated consistently, every energy would be -10.528057.

```
$ grep seed det_qmc_short.in.xml

$ export OMP_NUM_THREADS=1
$ mpirun -n 1 ../build_gccnewmpi/bin/qmcpack det_qmc_short.in.xml >& out; ../qmcpack/nexus/bin/qmca -q eV det_qmc_short.s*.scalar.dat
LocalEnergy Variance ratio
det_qmc_short series 0 -10.528057 +/- 0.123473 0.097440 +/- 0.012555 0.0093
$ cat multi2.txt
det_qmc_short.in.xml
det_qmc_short.in.xml

$ mpirun -n 2 ../build_gccnewmpi/bin/qmcpack multi2.txt >&out_m2; ../qmcpack/nexus/bin/qmca -q eV det_qmc_short.g*.s*.scalar.dat
LocalEnergy Variance ratio
det_qmc_short.g000 series 0 -10.399152 +/- 0.078468 0.448807 +/- 0.202805 0.0432
det_qmc_short.g001 series 0 -10.419865 +/- 0.195175 0.081891 +/- 0.038507 0.0079
$ rm *.g*.s*.dat
$ mpirun -n 16 ../build_gccnewmpi/bin/qmcpack multi16.txt >&out_m16; ../qmcpack/nexus/bin/qmca -q eV det_qmc_short.g*.s*.scalar.dat
LocalEnergy Variance ratio
det_qmc_short.g000 series 0 -10.480402 +/- 0.291461 0.309404 +/- 0.159051 0.0295
det_qmc_short.g001 series 0 -10.731790 +/- 0.100510 0.331670 +/- 0.089688 0.0309
det_qmc_short.g002 series 0 -10.297656 +/- 0.061974 0.083847 +/- 0.025331 0.0081
det_qmc_short.g003 series 0 -10.465141 +/- 0.058750 0.221925 +/- 0.147555 0.0212
det_qmc_short.g004 series 0 -10.397714 +/- 0.046317 0.113235 +/- 0.028581 0.0109
det_qmc_short.g005 series 0 -10.968746 +/- 0.212205 0.135360 +/- 0.055179 0.0123
det_qmc_short.g006 series 0 -10.377890 +/- 0.093169 0.079977 +/- 0.006489 0.0077
det_qmc_short.g007 series 0 -10.592697 +/- 0.224244 0.232637 +/- 0.109560 0.0220
det_qmc_short.g008 series 0 -10.369455 +/- 0.216723 0.129658 +/- 0.042167 0.0125
det_qmc_short.g009 series 0 -10.285464 +/- 0.470668 0.091856 +/- 0.037453 0.0089
det_qmc_short.g010 series 0 -10.332684 +/- 0.084356 0.067954 +/- 0.013056 0.0066
det_qmc_short.g011 series 0 -10.473175 +/- 0.158590 0.196747 +/- 0.073484 0.0188
det_qmc_short.g012 series 0 -10.357872 +/- 0.147707 0.151704 +/- 0.061738 0.0146
det_qmc_short.g013 series 0 -10.594037 +/- 0.037998 0.121146 +/- 0.055377 0.0114
det_qmc_short.g014 series 0 -10.682091 +/- 0.148414 0.121944 +/- 0.072871 0.0114
det_qmc_short.g015 series 0 -10.829192 +/- 0.157941 0.258428 +/- 0.175715 0.0239
```

Contributor guide

Open the contributing guide

Research direction

Start with the ensemble and batched-run tests associated with #4091 and #4093, then reproduce the independent, multi2.txt, and multi16.txt runs using det_qmc_short.in.xml and qmca. Compare the fixed-seed energies and define the desired seed semantics before implementation; done means the behavior is agreed, documented, and covered by deterministic ensemble checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, hpc
Domain
distributed-systems, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.