llnl / llnl/maestrowf

mpi4py bug

Open
#417 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
161
Forks
49
PR merge metrics
No merged PRs in 30d

Description

## Description
Hello,

I'm working on an LLNL project that uses maestro to manage ML workflows, and we've recently encountered an odd bug. If the following conditions are met, the job will hang indefinitely:
1. We pass a p-gen file to maestro that imports mpi4py directly or indirectly (through another imported module).
2. Maestro launches a job using more than one processor.
3. The job being launched also imports mpi4py.

## Reproducer

I've included files reproduce the issue below. There is 1 yaml file, 1 python script that maestro will launch with srun, and 3 parameter generation files. 1 of the parameter generation files works fine because it doesn't import mpi4py, and the other two parameter generation files will import mpi4py directly or indirectly and cause the job to hang.

Here are commands to reproduce each scenario:

This works: `maestro run -p param_gen.py mpi_bug.yaml`

This causes job hang: `maestro run -p mpi_param_gen.py mpi_bug.yaml`

This causes job hang: `maesturo run -p kosh_param_gen.py mpi_bug.yaml`

Files to reproduce:

`mpi_bug.yaml`:
```
batch:
bank: wbronze
host: rzgenie
queue: pdebug
type: slurm
description:
description: Reproduces mpi4py bug
name: bug_demo
env:
variables:
nodes: 1
procs: 4
walltime: '00:10:00'
script: /path/to/hello_world.py
study:
- description: Launch a simple script using srun
name: hello_world
run:
cmd: "#SBATCH --ntasks $(procs)\n\n $(LAUNCHER) python $(script)"
nodes: $(nodes)
procs: $(procs)
walltime: $(walltime)
```

`hello_world.py`:
```
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()

print(f"{rank}: hello world!")
```

`param_gen.py`:
```
from maestrowf.datastructures.core import ParameterGenerator

def get_custom_generator(*args, **kw_args):
p_gen = ParameterGenerator()
return p_gen
```

`mpi_param_gen.py`:
```
from maestrowf.datastructures.core import ParameterGenerator
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()

def get_custom_generator(*args, **kw_args):
p_gen = ParameterGenerator()
return p_gen
```

`kosh_param_gen.py`:
```
from maestrowf.datastructures.core import ParameterGenerator
#
# Kosh relies on mpi4py. This also causes maestro to hang.
#
import kosh

def get_custom_generator(*args, **kw_args):
p_gen = ParameterGenerator()
return p_gen
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mpi_bug.yaml, hello_world.py, and the three parameter-generation files, then run the listed maestro commands to compare the working and hanging cases. Trace how maestro loads parameter generators and launches the srun job; done means both mpi4py-importing reproductions complete without hanging while the non-MPI case still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.