Change to `--ntasks-per-socket` information for GPU jobs

What is inaccurate?

We assumed --ntasks-per-socket=1 in all cases. This works on amperenodes by coincidence. This does not work on pascalnodes when more than 2 GPUs are used.

--ntasks-per-socket is a maximum number of tasks per socket. Most modern node mainboards have one or two sockets. If --gres=gpu:3 is requested, then --ntasks-per-socket=1 will yield the following error from sbatch because only two of the tasks can be allocated, one for each of the two sockets.

Batch job submission failed: Requested node configuration is not available

The correct value to set is --ntasks-per-socket=(( ntasks / socketcount )), rounded up. In the above case this would give --gres=gpu:3 and --ntasks-per-socket=2. One socket would get two tasks, hit its maximum, and the other socket would get 1 task.

We will need to offer guidance on what the socket count is for each GPU partition and how to determine it. All of our current GPU nodes have two sockets. The number for a given node can be found with scontrol show node $nodeid | grep [^a-zA-Z]Sockets=. $nodeid can be obtained from scontrol show partition $partitionid | grep "[^a-zA-Z]Nodes=". The partitions are homogeneous so any node in the list is fine.

Where is the inaccuracy?

https://docs.rc.uab.edu/cheaha/slurm/gpu/#gpu-core-affinity-details