kubeflow / kubeflow/mpi-operator
mpijob using hostnetwork error
- Dominant language
- Go
- Stars
- 535
- Forks
- 238
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
hi, all
I want to use hostnetwork when submit mpijob to improve training performance. The yaml file is as below:
```
apiVersion: kubeflow.org/v1alpha2
kind: MPIJob
metadata:
annotations:
monitoring.netease.com/enable-grafana-dashboard: "true"
generateName: test-mpijob
generation: 2
labels:
fairing-deployer: mpijob
fairing-id: d7aaecf2-7e2e-11ea-8269-0a580ab29d87
kubeflow.netease.com/userid: huting3
namespace: ai-test
spec:
activeDeadlineSeconds: 3600
backoffLimit: 1
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
metadata:
annotations:
monitoring.netease.com/enable-grafana-dashboard: "true"
sidecar.istio.io/inject: "false"
creationTimestamp: null
labels:
fairing-deployer: mpijob
fairing-id: d7aaecf2-7e2e-11ea-8269-0a580ab29d87
kubeflow.netease.com/userid: huting3
name: fairing-deployer
spec:
hostNetwork: "true"
dnsPolicy: ClusterFirstWithHostNet
containers:
- command:
- mpirun
- --allow-run-as-root
- -np
- "2"
- -bind-to
- none
- -map-by
- slot
- -x
- NCCL_DEBUG=INFO
- -x
- LD_LIBRARY_PATH
- -x
- PATH
- -mca
- pml
- ob1
- -mca
- btl
- ^openib
- python
- /app/boot.py
env:
- name: FAIRING_RUNTIME
value: "1"
image: hub-inner.cn-east-p1.netease.com/deeplearning/fairing-job:8AB586D0
name: mpi
resources:
limits:
memory: 998579896320m
requests:
cpu: "1"
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /data
name: fairing-volume-data-huting3
workingDir: /app/
imagePullSecrets:
- name: hubinnercneastp1neteasecomdeeplearningstaffk8sai01serviceneteasecom
restartPolicy: Never
volumes:
- name: fairing-volume-data-huting3
persistentVolumeClaim:
claimName: data-huting3
Worker:
replicas: 2
template:
metadata:
annotations:
monitoring.netease.com/enable-grafana-dashboard: "true"
sidecar.istio.io/inject: "false"
labels:
fairing-deployer: mpijob
fairing-id: d7aaecf2-7e2e-11ea-8269-0a580ab29d87
kubeflow.netease.com/userid: huting3
name: fairing-deployer
spec:
hostNetwork: "true"
dnsPolicy: ClusterFirstWithHostNet
containers:
- env:
- name: FAIRING_RUNTIME
value: "1"
image: hub-inner.cn-east-p1.netease.com/deeplearning/fairing-job:8AB586D0
name: mpi
resources:
limits:
memory: 6002216796160m
nvidia.com/gpu: "1"
requests:
cpu: "4"
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /data
name: fairing-volume-data-huting3
workingDir: /app/
restartPolicy: Never
volumes:
- name: fairing-volume-data-huting3
persistentVolumeClaim:
claimName: data-huting3
slotsPerWorker: 1
```
but got error as below:
```
--------------------------------------------------------------------------
WARNING: Open MPI accepted a TCP connection from what appears to be a
another Open MPI process but cannot find a corresponding process
entry for that peer.
This attempted connection will be ignored; your MPI job may or may not
continue properly.
Local host: pri2-ainode28
PID: 30
--------------------------------------------------------------------------
```
what is the problem? any hints is good to me
Contributor guide
Research direction
No source file, test, or entry point is named. Start by reproducing the provided MPIJob manifest with hostNetwork enabled and inspect the operator-generated launcher and worker pods alongside the Open MPI warning. Done means identifying whether the operator or MPI configuration causes the connection mismatch and documenting a verified configuration or limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100