More flexible time-slicing strategy configuration

Note: this is a copy of https://gitlab.com/nvidia/kubernetes/device-plugin/-/issues/8

Issue or feature description

When a pod has requests/limits nvidia.com/gpu: 2, typically one slice per physical device is allocated, and the container gets two devices, as intended.

However, when multiple pods are created on the host (e.g., in a CI system launching a batch of tasks), it both slices might end up on a single device. In this case, the CUDA application in the container only sees a single device, and the device-to-device communication tests cannot run.

It would be helpful to have a setting that makes the one-slice-per-device distribution a requirement rather than a best-effort attempt when distributing slices between devices. For our current case, a global setting would work.

We can envision the case for having this configurable per-pod. While we currently only run small CI tasks on our k8s cluster (which are not performance-sensitive), we are contemplating having more performance testing. For it, the opposite behavior would be needed: being able to request all slots on a single device. That, however, is not a near-term problem for our project: likely, any performance testing will be done on a separate infrastructure.

Specific use case

• Small k8s cluster for running GitLab-based CI, with two NVIDIA T400 GPUs per node.
• CI tasks are pretty lightweight; time-slicing ("nvidia.com/gpu.replicas": "4") is used to utilize the hardware more efficiently.
• Three kinds of jobs: no GPU, single GPU, dual GPU. Latter require two different devices to test GPU-GPU communication (NVSHMEM, CUDA-aware MPI).
  • The number of devices can be checked by running nvidia-smi -L in the container.

For example, with a single node (2 GPUs, 4 slices each), allocating 1 then 5 then 2 slices triggers the problem for the last container:

$ cat test_nvidia5.yml 
apiVersion: v1
kind: Pod
metadata:
  name: al42and-nvidia-test5
spec:
  containers:
    - image: registry.gitlab.com/gromacs/gromacs/ci-ubuntu-20.04-gcc-11-cuda-11.7.1
      name: al42and-nvidia-test5
      command:
        - /bin/sh
        - "-c"
        - "sleep 24h"
      resources:
        requests:
          cpu: 2
          memory: 2Gi
          nvidia.com/gpu: 2
        limits:
          cpu: 4
          memory: 4Gi
          nvidia.com/gpu: 2
  restartPolicy: Never


$ kubectl create -f test_nvidia5.yml 
pod/al42and-nvidia-test5 created

$ kubectl exec al42and-nvidia-test5 -- nvidia-smi -L
GPU 0: NVIDIA T400 4GB (UUID: GPU-<.....>)

Discussion:

@klueska in the issue discussion on GitLab:

I'm not opposed to this in principle, but I don't know how to build support for this using the standard device plugin API.

The reason being that the scheduler won't know about this requirement and might decide to schedule the pod on a node that only has 2 GPU "slices" available on a single card. We can put a check at the node level to error out in case this ever happens, but the pod will then be stuck on the node in an error condition without the ability to be rescheduled.

To work around this, the pod can be wrapped in a deployment that will force rescheduling when the pod errors out, but it may end up thrashing if the scheduler keeps trying to put it on the same (unsuitable) node.

See also comments by @elezar at https://gitlab.com/gromacs/gromacs/-/merge_requests/3569#note_1439991699

Start by reviewing the Kubernetes device plugin API and the time-slicing configuration described in the issue. Reproduce the allocation sequence using the provided pod manifest, creating requests for 1, then 5, then 2 slices and checking each container with nvidia-smi -L. Done should mean the distribution behavior is configurable and the requested multi-device allocation is handled explicitly.