kubeflow / kubeflow/mpi-operator

MPIJobs with dots in name will lead to wrong hostfiles

Open
#733 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

Hello,

we noticed that mpirun will not run correctly when dots are used in MPIJob names.

For example
```yaml
apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
name: myjob.1
spec:
slotsPerWorker: 1
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
restartPolicy: OnFailure
replicas: 1
template:
spec:
containers:
- image: IMAGE
name: launcher
imagePullPolicy: Always
command:
- mpirun
- --allow-run-as-root
- -np
- "2"
Worker:
replicas: 2
template:
spec:
containers:
- image: IMAGE
name: worker
imagePullPolicy: Always
````

will lead to this error message when `mpirun` is executed:
```
A hostfile was provided that contains multiple definitions
of the slot count for at least one node:

hostfile: hosts
node: mpi-worker

You can either list a node multiple times, once for each slot,
or you can provide a single line that contains "slot=N". Mixing
the two methods is not supported.

Please correct the hostfile and try again.
```

In the image, OpenMPI v4 was installed.

I assume this is caused by how openmpi interprets the hostnames which will include the dot from the MPIJob name. Also see https://github.com/open-mpi/ompi/issues/4732#issuecomment-359446798 for a related discussion.

This can also lead to the case where the `mpirun` command runs successfully but only one worker is used.

Just mentioning it here as well in case someone stumbles upon this. We will probably validate the MPIJob name on creation or check different mpi settings like `-mca orte_keep_fqdn_hostnames t`.

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reproducing the MPIJob example with a dotted name and trace how worker hostnames and the hostfile are generated; done means dotted MPIJob names no longer produce conflicting slot definitions or silently use only one worker.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.