kubeflow / kubeflow/mpi-operator

Cannot run MPIJob

Open
#279 3 comments 1 reaction 0 assignees View on GitHub
area/engprod kind/bug
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

Trying out the example specified for v1alpha1 version: https://github.com/kubeflow/mpi-operator/blob/master/examples/v1alpha1/tensorflow-benchmarks-custom.yaml

```yaml
apiVersion: kubeflow.org/v1alpha1
kind: MPIJob
metadata:
name: tensorflow-benchmarks-16-custom
spec:
replicas: 4
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 4
```

The launcher pod gets into CrashLoopBackOff loop as the init container `kubectl-delivery` fails with error:

```
$ kubectl describe pod tensorflow-benchmarks-16-custom-launcher-kqnpd
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m4s default-scheduler Successfully assigned default/tensorflow-benchmarks-16-custom-launcher-kqnpd to ip-192-168-189-39.us-west-2.compute.internal
Normal Created 3m12s (x4 over 4m) kubelet, ip-192-168-189-39.us-west-2.compute.internal Created container kubectl-delivery
Normal Started 3m11s (x4 over 4m) kubelet, ip-192-168-189-39.us-west-2.compute.internal Started container kubectl-delivery
Warning BackOff 2m43s (x7 over 3m58s) kubelet, ip-192-168-189-39.us-west-2.compute.internal Back-off restarting failed container
Normal Pulling 2m29s (x5 over 4m3s) kubelet, ip-192-168-189-39.us-west-2.compute.internal Pulling image "mpioperator/kubectl-delivery:latest"
Normal Pulled 2m28s (x5 over 4m) kubelet, ip-192-168-189-39.us-west-2.compute.internal Successfully pulled image "mpioperator/kubectl-delivery:latest"
```

```
$ kubectl logs tensorflow-benchmarks-16-custom-launcher-kqnpd -c kubectl-delivery
I0707 17:45:11.131182 1 server.go:53] NAMESPACE not set, use default namespace
I0707 17:45:11.131227 1 server.go:63] Scoping operator to namespace default
I0707 17:45:11.131235 1 server.go:67] [API Version: v1 Version: v0.1.0 Git SHA: Not provided. Built: Not provided. Go Version: go1.13.6 Go OS/Arch: linux/amd64]
I0707 17:45:11.131256 1 server.go:70] Server options: &{Kubeconfig: MasterURL: Threadiness:2 PrintVersion:false Namespace:}
W0707 17:45:11.131410 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
F0707 17:45:11.132470 1 server.go:95] Error open file[/etc/mpi/hostfile]: open /etc/mpi/hostfile: no such file or directory
```

I am using the AWS EKS cluster v1.14

Need help in debugging the issue.

Contributor guide

Open the contributing guide

Research direction

Start with examples/v1alpha1/tensorflow-benchmarks-custom.yaml and reproduce the launcher failure on Kubernetes, then inspect the kubectl-delivery init container and its hostfile configuration. Done means the example creates a launcher pod whose init container does not enter CrashLoopBackOff and the MPIJob can run.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, tensorflow
Domain
devops, distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.