Evaluate integrating NVCRE into nvca cluster-validator for compute-plane GPU-fabric checks

Description

cluster-validator's compute-plane checks include a node_to_node connectivity probe, but it only tests generic CNI/overlay TCP reachability -- it never exercises the GPU interconnect (NVLink, InfiniBand, RoCE, EFA) that a real training job depends on. NVIDIA's OSS Cluster Readiness Engine (NVCRE, https://github.com/NVIDIA/cluster-readiness-engine) runs real NCCL/training workloads against a cluster and reports per-node GPU-fabric health with a reason.

Evaluate whether NVCRE can be adopted for compute-plane GPU-fabric burn-in, either as a standalone step or as an addition to cluster-validator's check set, without replacing cluster-validator's platform-config checks (webhooks, netpol, gateway routes, registry credentials, etc.), which are outside NVCRE's scope.

Definition of Done

• Written recommendation: is NVCRE viable as-is, and if so at what scope (new check category vs. separate scheduled burn-in vs. not adopted).
• If viable, a scoped integration plan: which NVCRE catalog entry/entries to trigger, sync vs. async invocation, new cluster dependencies (Kubeflow Trainer, Prometheus Operator CRDs), and RBAC needed.
• Any gaps in NVCRE that would need upstream contribution are called out explicitly, separate from gaps that are simply out of NVCRE's scope.
• Decision and plan reviewed with the compute-plane team before implementation work is filed as follow-up issues.

Resources (optional)

• NVCRE repository: https://github.com/NVIDIA/cluster-readiness-engine
• NVCRE docs: https://docs.nvidia.com/cluster-readiness-engine/overview/introduction/
• cluster-validator: src/compute-plane-services/nvca/internal/clustervalidator

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.