kubeflow / kubeflow/mpi-operator
MPIJobs with dots in name will lead to wrong hostfiles
- Dominant language
- Go
- Stars
- 535
- Forks
- 238
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
Hello,
we noticed that mpirun will not run correctly when dots are used in MPIJob names.
For example
```yaml
apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
name: myjob.1
spec:
slotsPerWorker: 1
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
restartPolicy: OnFailure
replicas: 1
template:
spec:
containers:
- image: IMAGE
name: launcher
imagePullPolicy: Always
command:
- mpirun
- --allow-run-as-root
- -np
- "2"
Worker:
replicas: 2
template:
spec:
containers:
- image: IMAGE
name: worker
imagePullPolicy: Always
````
will lead to this error message when `mpirun` is executed:
```
A hostfile was provided that contains multiple definitions
of the slot count for at least one node:
hostfile: hosts
node: mpi-worker
You can either list a node multiple times, once for each slot,
or you can provide a single line that contains "slot=N". Mixing
the two methods is not supported.
Please correct the hostfile and try again.
```
In the image, OpenMPI v4 was installed.
I assume this is caused by how openmpi interprets the hostnames which will include the dot from the MPIJob name. Also see https://github.com/open-mpi/ompi/issues/4732#issuecomment-359446798 for a related discussion.
This can also lead to the case where the `mpirun` command runs successfully but only one worker is used.
Just mentioning it here as well in case someone stumbles upon this. We will probably validate the MPIJob name on creation or check different mpi settings like `-mca orte_keep_fqdn_hostnames t`.
Contributor guide
Research direction
No source file or test is named. Start by reproducing the MPIJob example with a dotted name and trace how worker hostnames and the hostfile are generated; done means dotted MPIJob names no longer produce conflicting slot definitions or silently use only one worker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100