bug: `K8sNcclWorkload`: `nccl_allreduce_job.yaml` has no `/dev/shm` volume, causing intermittent UCX segfaults at higher local GPU/rank counts

Version

v0.12.0 (commit 6ab356d)

Platform

kubernetes

Describe the bug

isvtest/src/isvtest/workloads/manifests/k8s/nccl_allreduce_job.yaml (used by K8sNcclWorkload) does not mount a /dev/shm volume, so the container falls back to Kubernetes' default 64Mi tmpfs. With 8 local MPI ranks (one per GPU), OpenMPI/UCX's POSIX shared-memory transport needs more than 64Mi to bootstrap, and running out of space manifests as a segfault rather than a clean error:

mm_iface failed to allocate receive FIFO
...
[ucp_address_unpack] Caught signal 11 (Segmentation fault)

The sibling manifest nccl_allreduce_mpijob.yaml (used by K8sNcclMultiNodeWorkload) already mounts a correctly-sized /dev/shm:

volumes:
  - name: shm
    emptyDir:
      medium: Memory
      sizeLimit: 8Gi

...suggesting nccl_allreduce_job.yaml simply missed the same fix.

Minimum reproducible example

1. Deploy `nccl_allreduce_job.yaml` unmodified on an 8-GPU node (`nvidia.com/gpu: 8`, `-np 8`).
2. Observe the job fails (segfault) on most/all `backoffLimit` attempts.
3. Add the same `/dev/shm` volume/mount used in `nccl_allreduce_mpijob.yaml` and rerun — passes reliably (confirmed ~67 GB/s avg bus bandwidth across multiple consecutive runs on 8x A100).

Relevant log output

Other/Misc.

Suggested Fix

Add the following to nccl_allreduce_job.yaml's pod spec (mirroring nccl_allreduce_mpijob.yaml):

volumeMounts:
  - name: shm
    mountPath: /dev/shm
...
volumes:
  - name: shm
    emptyDir:
      medium: Memory
      sizeLimit: 8Gi

Additional Notes

• Ruled out LimitMEMLOCK/ulimit on the container runtime's systemd unit as a cause — raising it to  infinity` had no effect on the failure.

Code of Conduct

• I agree to follow AI Cloud Validation Suite's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report