MPIJob Pods shows status RUNNING despite MPIJob Completed

I am testing out TFJob and MPIJob operators from kubeflow/manifests-v1.1.0 branch on AWS EKS K8s=1.14. I am able to schedule TFJob and MPIJob successfully and these jobs also complete fine. I have verified it by watching the k8s events that I get MPIJob completed successfully.

However, I observe that the worker pods still keep on running even after the launcher pod and the MPI Job has completed. I see this behavior for TFJob as well.

I have applied the following yaml for the MPIJob.
```
apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
name: tf2-keras-mnist-mpi-gpu
spec:
slotsPerWorker: 1
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- image: docker.io/horovod/horovod:0.19.3-tf2.1.0-torch-mxnet1.6.0-py3.6-gpu
name: keras-mnist-mpi-launcher
command:
- mpirun
args:
- -np
- "2"
- --allow-run-as-root
- -bind-to
- none
- -map-by
- slot
- -x
- LD_LIBRARY_PATH
- -x
- PATH
- -mca
- pml
- ob1
- -mca
- btl
- ^openib
- python
- /examples/tensorflow2_keras_mnist.py
resources:
limits:
cpu: 1
memory: 2Gi
Worker:
replicas: 2
template:
spec:
containers:
- image: docker.io/horovod/horovod:0.19.3-tf2.1.0-torch-mxnet1.6.0-py3.6-gpu
name: keras-mnist-mpi-worker
resources:
limits:
nvidia.com/gpu: 1
```

I collected the mpi controller logs which clearly shows that the MPIJob is indeed completed but the controller is failing to sync that info.

**Observe the pods in my namespace:**
```
(base) asaha-mbp151:exploration asaha$ kubectl get pods -n asaha
NAME READY STATUS RESTARTS AGE
tf2-keras-mnist-mpi-gpu-launcher-zj6gv 0/1 Completed 0 16m
tf2-keras-mnist-mpi-gpu-worker-0 1/1 Running 0 16m
tf2-keras-mnist-mpi-gpu-worker-1 1/1 Running 0 16m
```

**Observe the mpi controller logs in kubeflow namespace:**
```
(base) asaha-mbp151:exploration asaha$ kubectl get pods -n kubeflow
NAME READY STATUS RESTARTS AGE
mpi-operator-5f7f4d94cc-qjzb9 1/1 Running 0 27m
(base) asaha-mbp151:exploration asaha$ kubectl logs -f mpi-operator-5f7f4d94cc-qjzb9 -n kubeflow
I1125 14:30:17.716033 1 server.go:88] Using cluster scoped operator
I1125 14:30:17.716068 1 server.go:94] [API Version: v1alpha2 Version: v0.2.2 Git SHA: 75f424a802dafb3662bc5c76b8f3c3cb60127fac Built: 2020-10-20 00:41:20 Go Version: go1.13.6 Go OS/Arch: linux/amd64]
I1125 14:30:17.716082 1 server.go:97] Server options: &{Kubeconfig: MasterURL: KubectlDeliveryImage:mpioperator/kubectl-delivery:latest Threadiness:2 MonitoringPort:0 PrintVersion:false GangSchedulingName: Namespace: LockNamespace:kubeflow}
W1125 14:30:17.716158 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1125 14:30:17.742811 1 leaderelection.go:235] attempting to acquire leader lease kubeflow/mpi-operator...
I1125 14:30:17.742832 1 server.go:204] Start listening to 8080 for health check
I1125 14:30:17.747630 1 server.go:253] New leader has been elected: mpi-operator-5f7f4d94cc-brxv8_9f52d006-afea-4705-98ac-1c7108b75828
I1125 14:30:34.722869 1 leaderelection.go:245] successfully acquired lease kubeflow/mpi-operator
I1125 14:30:34.722979 1 server.go:242] Leading started
I1125 14:30:34.722986 1 event.go:258] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kubeflow", Name:"mpi-operator", UID:"9315cf6d-2e88-11eb-8bf4-0645e9fd4c9d", APIVersion:"v1", ResourceVersion:"259432", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' mpi-operator-5f7f4d94cc-qjzb9_235a8483-1d7b-4634-a9cf-84476af9f050 became leader
I1125 14:30:34.723214 1 mpi_job_controller.go:227] Setting up event handlers
I1125 14:30:34.723301 1 mpi_job_controller.go:361] Starting MPIJob controller
I1125 14:30:34.723328 1 mpi_job_controller.go:364] Waiting for informer caches to sync
I1125 14:30:34.823445 1 mpi_job_controller.go:374] Starting workers
I1125 14:30:34.823471 1 mpi_job_controller.go:380] Started workers
I1125 14:39:21.748875 1 mpi_job_controller.go:455] Finished syncing job "asaha/tf2-keras-mnist-mpi-gpu" (118.401638ms)
E1125 14:39:21.748911 1 mpi_job_controller.go:442] error syncing 'asaha/tf2-keras-mnist-mpi-gpu': the server could not find the requested resource (put mpijobs.kubeflow.org tf2-keras-mnist-mpi-gpu)
I1125 14:41:04.270894 1 mpi_job_controller.go:455] Finished syncing job "asaha/tf2-keras-mnist-mpi-gpu" (1.746921ms)
E1125 14:41:04.270926 1 mpi_job_controller.go:442] error syncing 'asaha/tf2-keras-mnist-mpi-gpu': the server could not find the requested resource (put mpijobs.kubeflow.org tf2-keras-mnist-mpi-gpu)
I1125 14:43:48.113053 1 mpi_job_controller.go:455] Finished syncing job "asaha/tf2-keras-mnist-mpi-gpu" (1.998845ms)
E1125 14:43:48.113093 1 mpi_job_controller.go:442] error syncing 'asaha/tf2-keras-mnist-mpi-gpu': the server could not find the requested resource (put mpijobs.kubeflow.org tf2-keras-mnist-mpi-gpu)
I1125 14:54:43.473321 1 event.go:258] Event(v1.ObjectReference{Kind:"MPIJob", Namespace:"asaha", Name:"tf2-keras-mnist-mpi-gpu", UID:"018ee039-2f2c-11eb-8bf4-0645e9fd4c9d", APIVersion:"kubeflow.org/v1alpha2", ResourceVersion:"260900", FieldPath:""}): type: 'Normal' reason: 'MPIJobSucceeded' MPIJob asaha/tf2-keras-mnist-mpi-gpu successfully completed.
I1125 14:54:43.476006 1 mpi_job_controller.go:455] Finished syncing job "asaha/tf2-keras-mnist-mpi-gpu" (2.807611ms)
E1125 14:54:43.476037 1 mpi_job_controller.go:442] error syncing 'asaha/tf2-keras-mnist-mpi-gpu': the server could not find the requested resource (put mpijobs.kubeflow.org tf2-keras-mnist-mpi-gpu)
```

**Describe the MPIJob:**
```
kubectl describe mpijob tf2-keras-mnist-mpi-gpu -n asaha
Name: tf2-keras-mnist-mpi-gpu
Namespace: asaha
Labels:
Annotations:
API Version: kubeflow.org/v1
Kind: MPIJob
Metadata:
Creation Timestamp: 2020-11-25T14:39:21Z
Generation: 1
Resource Version: 260900
Self Link: /apis/kubeflow.org/v1/namespaces/asaha/mpijobs/tf2-keras-mnist-mpi-gpu
UID: 018ee039-2f2c-11eb-8bf4-0645e9fd4c9d
Spec:
Clean Pod Policy: Running
Mpi Replica Specs:
Launcher:
Replicas: 1
Template:
Spec:
Containers:
Args:
-np
2
--allow-run-as-root
-bind-to
none
-map-by
slot
-x
LD_LIBRARY_PATH
-x
PATH
-mca
pml
ob1
-mca
btl
^openib
python
/examples/tensorflow2_keras_mnist.py
Command:
mpirun
Image: docker.io/horovod/horovod:0.19.3-tf2.1.0-torch-mxnet1.6.0-py3.6-gpu
Name: keras-mnist-mpi-launcher
Resources:
Limits:
Cpu: 1
Memory: 2Gi
Worker:
Replicas: 2
Template:
Spec:
Containers:
Image: docker.io/horovod/horovod:0.19.3-tf2.1.0-torch-mxnet1.6.0-py3.6-gpu
Name: keras-mnist-mpi-worker
Resources:
Limits:
nvidia.com/gpu: 1
Slots Per Worker: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal MPIJobSucceeded 100s (x3 over 35m) mpi-job-controller MPIJob asaha/tf2-keras-mnist-mpi-gpu successfully completed.
```

Start in mpi_job_controller.go at the sync path reported in the controller logs, and compare its update request with the kubeflow.org/v1 MPIJob resource shown. Reproduce with the supplied manifest and inspect whether the failed resource update is resolved. Done means the completed MPIJob status is synchronized and worker pod cleanup follows the configured policy.

Written by the indexing model from the issue text.