nvsnap: multi-node checkpoint/restore blocked by RDMA device mappings (UCX_TLS workaround is single-node only)

Problem

criu-v2 cannot dump a process that has UCX InfiniBand transports active. Two
distinct failures, both reproduced on TensorRT-LLM and NIM images (each ships
HPC-X, so UCX initializes even at TP=1):

Error (criu/proc_parse.c:121): handle_device_vma plugin failed

UCX's ib transport maps doorbell pages from /dev/infiniband/uverbsN. No
CRIU plugin claims that device, so mapping collection aborts. The containers
are privileged, so they see the node's real RDMA devices regardless of whether
the workload needs them.

Error (criu/cr-dump.c:511): Task N with SysVIPC shmem map @... doesn't live in IPC ns

UCX's shared-memory transport allocates a SysV segment alongside its POSIX one.
criu-v2 nsenters into the container's IPC namespace, so CRIU sees no ipc ns to
dump and rejects the mapping.

Current workaround

Both single-GPU workloads pin an explicit UCX transport allowlist that omits
ib and sysv:

- name: UCX_TLS
  value: "tcp,self,posix,cma,cuda_copy,cuda_ipc"

This is a positive allowlist rather than an exclusion (^ib,sysv) because the
exclusion syntax resolves through UCX alias definitions that vary by version.
Note that the sm alias is not sufficient: it enables both posix and sysv.

Applied in deploy/k8s/workloads/trtllm-small.yaml and
deploy/k8s/workloads/nim-llama-8b.yaml. Measured cost on a single-node,
single-GPU pod is not detectable (inference time 1m33s before, 1m32s across
three runs after). posix keeps shared memory working and cma keeps the fast
intra-node path, and a single-node pod has no RDMA peer to reach.

Why this does not generalize

The workaround trades away RDMA, which is only free because these workloads are
single-node. It must not be promoted into the admission webhook or a
cluster-wide default. On any genuine multi-node deployment (multi-node TP/PP,
disaggregated prefill/decode, NCCL or UCC over UCX) removing ib forces
inter-node traffic onto tcp, which is a serious throughput and latency
regression. RDMA is the point of those topologies.

So today: multi-node checkpoint/restore has no path. Either the workload keeps
ib and cannot be dumped, or it drops ib and is no longer a viable multi-node
deployment.

What a real fix looks like

Handle the device mappings rather than avoiding them. Two candidate directions:

1. A CRIU plugin that claims /dev/infiniband/* via HANDLE_DEVICE_VMA and
   re-establishes the mapping on restore. Mirrors what the CUDA plugin already
   does for /dev/nvidia*.
2. Cooperative teardown and re-establish of UCX around the checkpoint, the way
   cuda-checkpoint releases GPU state and closes device fds before CRIU
   collects descriptors. Queue pair state is connection state, so this likely
   has to be coordinated across all ranks, not per-process.

Option 2 is structurally the same problem as the multi-GPU peer-state work: the
resource is not just a mapping, it is a live connection to a peer, and severing
it transparently is the hard part.

Related in kind: #465 (gdrcopy /dev/gdrdrv fd) was the same class of failure
and was resolved by teaching the CUDA plugin to claim the device fd.

Acceptance criteria

• A multi-node workload with ib transports enabled can be checkpointed and
  restored without a UCX_TLS override.
• Post-restore inter-node bandwidth is measured and comparable to pre-checkpoint.
• The UCX_TLS overrides can be removed from the single-node manifests, or the
  decision to keep them is documented as a deliberate single-node optimization.

Out of scope

Removing the current single-node workaround before a replacement exists. It is
correct for the workloads it is applied to.