questions about applying for nodes and gpus

Hi, i have been using mpi-operator to achieve distributed training recently。
the most command i used is “kubectl apply -f yaml”. Let me take the mpi-operator yaml for example
apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
name: cifar
spec:
slotsPerWorker: 1
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
restartPolicy: Never
template:
spec:
nodeName:
containers:
- image: 10.252.39.13:5000/deepspeed_ms:v2
name: mpijob-cifar-deepspeed-container
imagePullPolicy: Always
command:
- mpirun
- --allow-run-as-root
- python
- cifar/cifar10_deepspeed.py
- --epochs=100
- --deepspeed_mpi
- --deepspeed
- --deepspeed_config
- cifar/ds_config.json
env:
- name: OMP_NUM_THREADS
value: "1"
Worker:
replicas: 2
restartPolicy: OnFailure
template:
spec:
nodeName:
containers:
- image: 10.252.39.13:5000/deepspeed_ms:v2
name: deepspeed-mpijob-container
resources:
limits:
cpu: 2
memory: 8Gi
nvidia.com/gpu: 2
there are some questions i'm confused about:
1. the content about applying for gpu-resources seems in "Worker". Does the cifar-worker-0 and cifar-worker-1 pods are separatly applying for an node(in k8s cluster) with 2 gpu? Then what role does "slotsPerWorker" play?
2. I have excuted the "apply -f yaml" on the example yaml, with different replicas like "replicas: 1" ,"replicas: 4", and the resources limits was fixed at "nvidia.com/gpu: 1". I found interesting results :
*When replicas is set to large numerber, It takes a bit more time for the cifar-launcher pod to complete.
*the logs printed in cifar-launcher pod (when replicas: 4) were just like the result ( when replicas: 1) repeated 4 times.
so does these mean, the four pods have separately applyed for one gpu (from node in k8s cluster, and preferentially from the same node if gpus are enough), and printed out the average result. the whole process had nothing to do with distribution？
*by the way, when setting "repicas: 3" , there is error reported in my case:
train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size 64 !=21 * 1 * 3
this did confuse me.
4. If i have node-A with 1 gpus and node-B with 3 gpus, and wanna apply for 4 gpus, then how should i modify the "Worker" part?
Thank in advance for your apply~

The issue names the MPIJob YAML, Worker and Launcher sections, cifar/cifar10_deepspeed.py, and cifar/ds_config.json. Start by reading the mpi-operator MPIJob examples and Kubernetes GPU scheduling documentation, then compare launcher output and worker pod allocation for each replica setting. Done means documenting slotsPerWorker, replica/resource semantics, the node-A/node-B allocation case, and the batch-size error.

Written by the indexing model from the issue text.