aws-samples / aws-samples/aws-do-eks

MPIJob EFA example doesn't apply

Open
#19 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
68
Forks
35
Avg merge
4h 23m
Merged PRs (30d)
40

Description

The MPIJob EFA example [here](https://github.com/aws-samples/aws-do-eks/blob/7a0a1f679ecc0c752a0abea889f7a8d2ee95cfe3/Container-Root/eks/deployment/efa-device-plugin/test-nccl-efa-mount.yaml#L15), doesn't apply cleanly, it shows the following error:

```
Error from server (BadRequest): error when creating "mpijob.yaml": MPIJob in version "v2beta1" cannot be handled as a MPIJob: strict decoding error: unknown field "spec.mpiReplicaSpecs.launcher.template.spec.imagePullPolicy", unknown field "spec.mpiReplicaSpecs.worker.template.spec.imagePullPolicy"
```

The issue is that the `imagePullPolicy` must be specified on the container, not the spec. Changing it so the scheduler reads like this (and the same for the worker) allows it to apply:

```yaml
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
restartPolicy: OnFailure
containers:
#- image: .dkr.ecr.us-west-2.amazonaws.com/cuda-efa-nccl-tests:ubuntu18.04
- image: public.ecr.aws/w6p6i9i7/aws-efa-nccl-rdma:base-cudnn8-cuda11-ubuntu18.04
imagePullPolicy: IfNotPresent
```

Edit: actually, even with this fix, I'm unable to get it running. The connection from the launcher is refused by the worker: `Connection reset by 172.17.5.245 port 22`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.