kubeflow / kubeflow/mpi-operator
Cannot run MPIJob
- Dominant language
- Go
- Stars
- 535
- Forks
- 238
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
Trying out the example specified for v1alpha1 version: https://github.com/kubeflow/mpi-operator/blob/master/examples/v1alpha1/tensorflow-benchmarks-custom.yaml
```yaml
apiVersion: kubeflow.org/v1alpha1
kind: MPIJob
metadata:
name: tensorflow-benchmarks-16-custom
spec:
replicas: 4
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 4
```
The launcher pod gets into CrashLoopBackOff loop as the init container `kubectl-delivery` fails with error:
```
$ kubectl describe pod tensorflow-benchmarks-16-custom-launcher-kqnpd
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m4s default-scheduler Successfully assigned default/tensorflow-benchmarks-16-custom-launcher-kqnpd to ip-192-168-189-39.us-west-2.compute.internal
Normal Created 3m12s (x4 over 4m) kubelet, ip-192-168-189-39.us-west-2.compute.internal Created container kubectl-delivery
Normal Started 3m11s (x4 over 4m) kubelet, ip-192-168-189-39.us-west-2.compute.internal Started container kubectl-delivery
Warning BackOff 2m43s (x7 over 3m58s) kubelet, ip-192-168-189-39.us-west-2.compute.internal Back-off restarting failed container
Normal Pulling 2m29s (x5 over 4m3s) kubelet, ip-192-168-189-39.us-west-2.compute.internal Pulling image "mpioperator/kubectl-delivery:latest"
Normal Pulled 2m28s (x5 over 4m) kubelet, ip-192-168-189-39.us-west-2.compute.internal Successfully pulled image "mpioperator/kubectl-delivery:latest"
```
```
$ kubectl logs tensorflow-benchmarks-16-custom-launcher-kqnpd -c kubectl-delivery
I0707 17:45:11.131182 1 server.go:53] NAMESPACE not set, use default namespace
I0707 17:45:11.131227 1 server.go:63] Scoping operator to namespace default
I0707 17:45:11.131235 1 server.go:67] [API Version: v1 Version: v0.1.0 Git SHA: Not provided. Built: Not provided. Go Version: go1.13.6 Go OS/Arch: linux/amd64]
I0707 17:45:11.131256 1 server.go:70] Server options: &{Kubeconfig: MasterURL: Threadiness:2 PrintVersion:false Namespace:}
W0707 17:45:11.131410 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
F0707 17:45:11.132470 1 server.go:95] Error open file[/etc/mpi/hostfile]: open /etc/mpi/hostfile: no such file or directory
```
I am using the AWS EKS cluster v1.14
Need help in debugging the issue.
Contributor guide
Research direction
Start with examples/v1alpha1/tensorflow-benchmarks-custom.yaml and reproduce the launcher failure on Kubernetes, then inspect the kubectl-delivery init container and its hostfile configuration. Done means the example creates a launcher pod whose init container does not enter CrashLoopBackOff and the MPIJob can run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, tensorflow
- Domain
- devops, distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100