Nexus: MPI oversubscription not caught in a ws16 run
- Dominant language
- C++
- Stars
- 403
- Forks
- 154
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 82
Description
**Describe the bug**
Discovered while testing recent Nexus updates.
The recently added error handling presumably needs a slight expansion(?); I was surprised this was not caught. e.g. Abort if "not enough slots available" is found at the top of the *.err. Hopefully it is a one line fix.
Anyone running Nexus example with ws16 on their laptop is likely to hit this unless they have configured their MPI appropriately, since most people have <16 cores. Nexus should flag the error. Instead it waits for a non-existent process to complete or specific output to appear (?).
```
01_diamond_scf pk7$ ./diamond_lda.py --progress_tty
_____________________________________________________
Nexus 2.4.9
(c) Copyright 2012- Nexus developers
Please cite:
J. T. Krogel Comput. Phys. Commun. 198 154 (2016)
https://doi.org/10.1016/j.cpc.2015.08.012
_____________________________________________________
Checking current machine for Nexus dependencies...
Currently Available Nexus Dependencies:
Python = 3.14.5
numpy = 2.4.6 (required)
scipy = 1.17.1 (optional)
h5py = 3.16.0 (optional)
matplotlib = 3.10.9 (optional)
spglib = Unavailable (optional)
cif2cell = Unavailable (optional)
pydot = Unavailable (optional)
seekpath = Unavailable (optional)
Recommended Nexus Dependencies:
Python >= 3.10.0
numpy >= x.x.x (required)
scipy >= x.x.x (optional)
h5py >= x.x.x (optional)
matplotlib >= x.x.x (optional)
spglib >= x.x.x (optional)
cif2cell >= x.x.x (optional)
pydot >= x.x.x (optional)
seekpath >= x.x.x (optional)
Required dependencies are met,
however some optional dependencies are missing.
Some features of Nexus may be unavailable.
Missing dependencies:
- seekpath (optional)
- cif2cell (optional)
- pydot (optional)
- spglib (optional)
Applying user settings
Project starting
checking for file collisions
loading cascade images
cascade 0 checking in
checking cascade dependencies
all simulation dependencies satisfied
starting runs:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering ./runs/diamond/scf 0 MB
Executing:
export OMP_NUM_THREADS=1
mpirun -np 16 pw.x -input scf.in
```
(Waited 30 mins before interrupting)
```
$ cat runs/diamond/scf/scf.err
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 16
slots that were requested by the application:
pw.x
Either request fewer procs for your application, or make more slots
available for use.
A "slot" is the PRRTE term for an allocatable unit where we can
launch a process. The number of slots available are defined by the
environment in which PRRTE processes are run:
1. Hostfile, via "slots=N" clauses (N defaults to number of
processor cores if not provided)
2. The --host command line parameter, via a ":N" suffix on the
hostname (N defaults to 1 if not provided)
3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
4. If none of a hostfile, the --host command line parameter, or an
RM is present, PRRTE defaults to the number of processor cores
In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.
Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------
```
scf.out exists and is zero length.
**To Reproduce**
Develop.
```
cd nexus/nexus/examples/quantum_espresso/01_diamond_scf/
export PYTHONPATH=$HOMEprojects/qmc/git_QMCPACK_prckent/qmcpack/nexus
export PATH=$PATH:$HOME/projects/qmc/git_QMCPACK_prckent/qmcpack/nexus/nexus/bin
./diamond_lda.py --progress_tty
```
**Expected behavior**
Catch the failed job
**System:**
macOS laptop, macports dependencies
Contributor guide
Research direction
Reproduce the failure from nexus/examples/quantum_espresso/01_diamond_scf/diamond_lda.py with --progress_tty and inspect runs/diamond/scf/scf.err alongside the empty scf.out. Trace the Nexus run-monitoring entry point that handles job errors; done means the MPI “not enough slots available” failure is reported instead of waiting indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- hpc, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100