QMCPACK / QMCPACK/qmcpack

Nexus: MPI oversubscription not caught in a ws16 run

Open
#6,203 0 comments 0 reactions 0 assignees View on GitHub
bug nexus
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 12h
Merged PRs (30d)
82

Description

**Describe the bug**
Discovered while testing recent Nexus updates.

The recently added error handling presumably needs a slight expansion(?); I was surprised this was not caught. e.g. Abort if "not enough slots available" is found at the top of the *.err. Hopefully it is a one line fix.

Anyone running Nexus example with ws16 on their laptop is likely to hit this unless they have configured their MPI appropriately, since most people have <16 cores. Nexus should flag the error. Instead it waits for a non-existent process to complete or specific output to appear (?).

```
01_diamond_scf pk7$ ./diamond_lda.py --progress_tty

_____________________________________________________

Nexus 2.4.9

(c) Copyright 2012- Nexus developers

Please cite:
J. T. Krogel Comput. Phys. Commun. 198 154 (2016)
https://doi.org/10.1016/j.cpc.2015.08.012
_____________________________________________________

Checking current machine for Nexus dependencies...

Currently Available Nexus Dependencies:
Python = 3.14.5
numpy = 2.4.6 (required)
scipy = 1.17.1 (optional)
h5py = 3.16.0 (optional)
matplotlib = 3.10.9 (optional)
spglib = Unavailable (optional)
cif2cell = Unavailable (optional)
pydot = Unavailable (optional)
seekpath = Unavailable (optional)

Recommended Nexus Dependencies:
Python >= 3.10.0
numpy >= x.x.x (required)
scipy >= x.x.x (optional)
h5py >= x.x.x (optional)
matplotlib >= x.x.x (optional)
spglib >= x.x.x (optional)
cif2cell >= x.x.x (optional)
pydot >= x.x.x (optional)
seekpath >= x.x.x (optional)

Required dependencies are met,
however some optional dependencies are missing.
Some features of Nexus may be unavailable.

Missing dependencies:
- seekpath (optional)
- cif2cell (optional)
- pydot (optional)
- spglib (optional)

Applying user settings

Project starting
checking for file collisions
loading cascade images
cascade 0 checking in
checking cascade dependencies
all simulation dependencies satisfied

starting runs:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering ./runs/diamond/scf 0 MB
Executing:
export OMP_NUM_THREADS=1
mpirun -np 16 pw.x -input scf.in

```
(Waited 30 mins before interrupting)

```
$ cat runs/diamond/scf/scf.err
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 16
slots that were requested by the application:

pw.x

Either request fewer procs for your application, or make more slots
available for use.

A "slot" is the PRRTE term for an allocatable unit where we can
launch a process. The number of slots available are defined by the
environment in which PRRTE processes are run:

1. Hostfile, via "slots=N" clauses (N defaults to number of
processor cores if not provided)
2. The --host command line parameter, via a ":N" suffix on the
hostname (N defaults to 1 if not provided)
3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
4. If none of a hostfile, the --host command line parameter, or an
RM is present, PRRTE defaults to the number of processor cores

In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------
```

scf.out exists and is zero length.

**To Reproduce**

Develop.
```
cd nexus/nexus/examples/quantum_espresso/01_diamond_scf/
export PYTHONPATH=$HOMEprojects/qmc/git_QMCPACK_prckent/qmcpack/nexus
export PATH=$PATH:$HOME/projects/qmc/git_QMCPACK_prckent/qmcpack/nexus/nexus/bin
./diamond_lda.py --progress_tty
```

**Expected behavior**
Catch the failed job

**System:**
macOS laptop, macports dependencies

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure from nexus/examples/quantum_espresso/01_diamond_scf/diamond_lda.py with --progress_tty and inspect runs/diamond/scf/scf.err alongside the empty scf.out. Trace the Nexus run-monitoring entry point that handles job errors; done means the MPI “not enough slots available” failure is reported instead of waiting indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
hpc, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.