MPI plugin only sets ORTE env vars, so multi-node jobs silently run single-node on Open MPI 5
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
### What happened?
The MPI plugin configures OpenMPI through four ORTE-era MCA environment variables
(`pkg/constants/constants.go`):
```
OMPI_MCA_orte_default_hostfile
OMPI_MCA_orte_keep_fqdn_hostnames
OMPI_MCA_orte_set_default_slots
OMPI_MCA_plm_rsh_args
```
Open MPI 5.0 removed ORTE and replaced it with PRTE, so these are no longer recognised.
Open MPI ignores unknown `OMPI_MCA_*` variables silently, so on an Open MPI >= 5 image the
hostfile the plugin generates is never read. `mpirun` falls back to localhost and starts
one rank per core on the launcher pod, and the worker pods sit idle.
The job exits 0 and the TrainJob reports `Complete=True`, so nothing indicates that a
3-node job ran on 1 node. I only noticed because wall-clock time did not change when I
added nodes.
**Reproduction.** Image `mpioperator/mpi-pi:openmpi`, which is OpenMPI 5.0.7. The runtime
below is the `mpi-distributed` runtime that was removed in v2.1.0 by #2760, restored, with
`dependsOn` added so the launcher waits for the workers:
```yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
name: mpi-distributed
spec:
mlPolicy:
numNodes: 1
mpi:
numProcPerNode: 1
mpiImplementation: OpenMPI
sshAuthMountPath: /home/mpiuser/.ssh
runLauncherAsNode: true
template:
spec:
network:
publishNotReadyAddresses: true
successPolicy:
operator: All
targetReplicatedJobs: [launcher]
replicatedJobs:
- name: node
template:
spec:
template:
spec:
containers:
- name: node
image: mpioperator/mpi-pi:openmpi
securityContext: {runAsUser: 1000}
command: ["/usr/sbin/sshd"]
args: ["-De", "-f", "/home/mpiuser/.sshd_config"]
readinessProbe:
tcpSocket: {port: 2222}
initialDelaySeconds: 5
- name: launcher
dependsOn:
- name: node
status: Ready
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- name: node
image: mpioperator/mpi-pi:openmpi
securityContext: {runAsUser: 1000}
```
```yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
name: ompi5-repro
spec:
runtimeRef:
name: mpi-distributed
trainer:
numNodes: 3
numProcPerNode: 1
command:
- sh
- -c
- |
echo "=== ompi version ==="; mpirun --version 2>&1 | head -1
echo "=== hostfile ==="; cat /etc/mpi/hostfile
echo "=== MCA env ==="; env | grep -E '^(OMPI|PRTE)_' | sort
echo "=== mpirun, no explicit --hostfile ==="
mpirun /home/mpiuser/pi
```
`numNodes: 3`, `numProcPerNode: 1`, `runLauncherAsNode: true`, so the expected result is 3
ranks, one per pod. Launcher output:
```
=== ompi version ===
mpirun (Open MPI) 5.0.7
=== hostfile ===
ompi5-repro-node-0-0.ompi5-repro slots=1
ompi5-repro-node-0-1.ompi5-repro slots=1
ompi5-repro-launcher-0-0.ompi5-repro slots=1
=== MCA env ===
OMPI_MCA_orte_default_hostfile=/etc/mpi/hostfile
OMPI_MCA_orte_keep_fqdn_hostnames=true
OMPI_MCA_orte_set_default_slots=1
OMPI_MCA_plm_rsh_args=-o ConnectionAttempts=10
=== mpirun, no explicit --hostfile ===
Workers: 20
Rank 0 on host ompi5-repro-launcher-0-0
Rank 1 on host ompi5-repro-launcher-0-0
... (all 20 on the launcher)
pi is approximately 3.1414761599999999
```
TrainJob status: `Complete=True`.
The hostfile is correct and mounted, it is just not read. 20 is the core count of the node
the launcher landed on, so the rank count varies with scheduling, which makes this harder
to spot. Passing `mpirun --hostfile /etc/mpi/hostfile` explicitly gives the expected 3
ranks across 3 pods, confirming the hostfile is fine and only the env var is ignored.
Two checks from inside that image:
```
$ ompi_info --all --parsable | grep -c orte_
0
$ prte_info --all | grep -E 'prte_default_hostfile|prte_keep_fqdn|prte_set_default_slots|plm_ssh'
MCA prte: "prte_default_hostfile"
MCA prte: "prte_keep_fqdn_hostnames"
MCA prte: "prte_set_default_slots"
MCA plm: "plm_ssh_agent"
MCA plm: "plm_ssh_args"
```
So the ORTE parameter namespace is gone entirely, not aliased.
**Affected versions.** The four constants are byte-identical in v2.1.0, v2.2.1, v2.3.0 and
master, and `git grep -i prte` returns nothing anywhere in the tree. Runtimes shipping
Open MPI 4 images are unaffected ( `deepspeed-distributed` is 4.1.2) which is presumably
why this has not surfaced in CI.
### What did you expect to happen?
`numNodes: 3` with `numProcPerNode: 1` should produce 3 ranks, one per pod, regardless of
whether the runtime image ships Open MPI 4 or 5. Failing that, it should fail loudly rather
than silently running single-node and reporting success.
**Suggested fix.** Emit both variable families rather than detecting a version. Open MPI 4
ignores `PRTE_MCA_*` and Open MPI 5 ignores `OMPI_MCA_orte_*`, so a single runtime stays
correct on both and no API change is needed.
This is the main reason I would prefer it to a version field on `mpiImplementation`: we can
keep one OpenMPI runtime that works regardless of which Open MPI version a user's image
ships, instead of maintaining one runtime per MPI version. We would also like a single
MPICH runtime once MPICH is supported (#4025). Happy to help with that work if useful.
| current (ORTE) | added (PRTE) |
|---|---|
| `OMPI_MCA_orte_default_hostfile` | `PRTE_MCA_prte_default_hostfile` |
| `OMPI_MCA_orte_keep_fqdn_hostnames` | `PRTE_MCA_prte_keep_fqdn_hostnames` |
| `OMPI_MCA_orte_set_default_slots` | `PRTE_MCA_prte_set_default_slots` |
| `OMPI_MCA_plm_rsh_args` | `PRTE_MCA_plm_ssh_args` |
One caveat that cost me a debugging round: `prte_keep_fqdn_hostnames` must be set to
**false**, not mirrored from the ORTE value. The generated hostfile uses JobSet's
short-form Pod DNS name (`.`), which never string-matches what PRTE resolves
for itself. With it enabled and `runLauncherAsNode: true`, PRTE treats the launcher's own
hostfile entry as a remote host and tries to SSH to itself:
```
ssh: connect to host ompi5-repro-launcher-0-0.ompi5-repro port 2222: Connection refused
PRTE has lost communication with a remote daemon.
```
ORTE tolerates the same mismatch, PRTE does not.
With the fix applied: same manifests, same cluster, only the controller image swapped:
```
=== MCA env ===
OMPI_MCA_orte_default_hostfile=/etc/mpi/hostfile
OMPI_MCA_orte_keep_fqdn_hostnames=true
OMPI_MCA_orte_set_default_slots=1
OMPI_MCA_plm_rsh_args=-o ConnectionAttempts=10
PRTE_MCA_plm_ssh_args=-o ConnectionAttempts=10
PRTE_MCA_prte_default_hostfile=/etc/mpi/hostfile
PRTE_MCA_prte_keep_fqdn_hostnames=false
PRTE_MCA_prte_set_default_slots=1
=== mpirun, no explicit --hostfile ===
Workers: 3
Rank 0 on host ompi5-repro-launcher-0-0
Rank 1 on host ompi5-repro-node-0-0
Rank 2 on host ompi5-repro-node-0-1
```
I have this implemented and running on our staging cluster: about 16 lines in
`pkg/runtime/framework/plugins/mpi/mpi.go` plus four constants, existing unit tests
updated, and the new names added to `MPIReservedEnvNames`. Happy to open a PR if the
approach looks right.
Related: kubeflow/mpi-operator#842 reports the same root cause in mpi-operator and is still
open.
### Environment
Kubernetes version:
```bash
$ kubectl version
Client Version: v1.35.1
Kustomize Version: v5.7.1
Server Version: v1.30.5
```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow-system -l app.kubernetes.io/name=kubeflow-trainer -o jsonpath="{.items[*].spec.containers[*].image}"
```
(returns nothing on v2.1.0 and the pod label is `app.kubernetes.io/name=trainer`)
```bash
$ kubectl get pods -n kubeflow-system -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"
ghcr.io/kubeflow/trainer/trainer-controller-manager:v2.1.0
```
JobSet v0.10.1. Reproduced on stock v2.1.0; the cluster now runs a locally patched build
of the same version carrying the fix above.
Kubeflow Python SDK version: not used since TrainJobs are applied directly as YAML
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Contributor guide
Assessment
This issue has not been assessed yet.