kubeflow / kubeflow/mpi-operator

Launcher has been unable to complete initialization

Open
#481 25 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

Hello, I tried to run the mpi-operator/examples/v1/horovod/tensorflow-mnist-elastic.yaml case, and found that the launcher has been unable to complete the Init, the log shows the following

# kubectl get pods -o wide -n mpi-operator
```
tensorflow-mnist-elastic-launcher 0/1 Init:0/1 0 6m11s
tensorflow-mnist-elastic-worker-0 1/1 Running 0 6m11s
tensorflow-mnist-elastic-worker-1 1/1 Running 0 6m11s
```

# kubectl logs tensorflow-mnist-elastic-launcher -n mpi-operator
```
Error from server (BadRequest): container "mpi-launcher" in pod "tensorflow-mnist-elastic-launcher" is waiting to start: PodInitializing
```

How can I troubleshoot or solve

Contributor guide

Open the contributing guide

Research direction

The reproduction is mpi-operator/examples/v1/horovod/tensorflow-mnist-elastic.yaml; start by running it and inspecting the launcher's init container status and pod events with kubectl. Trace why the launcher remains at Init:0/1 while workers run, then verify the example reaches initialization and document the troubleshooting steps or fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, tensorflow
Domain
distributed-systems, infrastructure, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.