mpi4py bug
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 161
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
## Description
Hello,
I'm working on an LLNL project that uses maestro to manage ML workflows, and we've recently encountered an odd bug. If the following conditions are met, the job will hang indefinitely:
1. We pass a p-gen file to maestro that imports mpi4py directly or indirectly (through another imported module).
2. Maestro launches a job using more than one processor.
3. The job being launched also imports mpi4py.
## Reproducer
I've included files reproduce the issue below. There is 1 yaml file, 1 python script that maestro will launch with srun, and 3 parameter generation files. 1 of the parameter generation files works fine because it doesn't import mpi4py, and the other two parameter generation files will import mpi4py directly or indirectly and cause the job to hang.
Here are commands to reproduce each scenario:
This works: `maestro run -p param_gen.py mpi_bug.yaml`
This causes job hang: `maestro run -p mpi_param_gen.py mpi_bug.yaml`
This causes job hang: `maesturo run -p kosh_param_gen.py mpi_bug.yaml`
Files to reproduce:
`mpi_bug.yaml`:
```
batch:
bank: wbronze
host: rzgenie
queue: pdebug
type: slurm
description:
description: Reproduces mpi4py bug
name: bug_demo
env:
variables:
nodes: 1
procs: 4
walltime: '00:10:00'
script: /path/to/hello_world.py
study:
- description: Launch a simple script using srun
name: hello_world
run:
cmd: "#SBATCH --ntasks $(procs)\n\n $(LAUNCHER) python $(script)"
nodes: $(nodes)
procs: $(procs)
walltime: $(walltime)
```
`hello_world.py`:
```
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
print(f"{rank}: hello world!")
```
`param_gen.py`:
```
from maestrowf.datastructures.core import ParameterGenerator
def get_custom_generator(*args, **kw_args):
p_gen = ParameterGenerator()
return p_gen
```
`mpi_param_gen.py`:
```
from maestrowf.datastructures.core import ParameterGenerator
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
def get_custom_generator(*args, **kw_args):
p_gen = ParameterGenerator()
return p_gen
```
`kosh_param_gen.py`:
```
from maestrowf.datastructures.core import ParameterGenerator
#
# Kosh relies on mpi4py. This also causes maestro to hang.
#
import kosh
def get_custom_generator(*args, **kw_args):
p_gen = ParameterGenerator()
return p_gen
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with mpi_bug.yaml, hello_world.py, and the three parameter-generation files, then run the listed maestro commands to compare the working and hanging cases. Trace how maestro loads parameter generators and launches the srun job; done means both mpi4py-importing reproductions complete without hanging while the non-MPI case still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100