Validate preferred and fallback cluster shapes for Rubin BF16 preferred-cluster GEMM

Summary

PersistentDenseGemmKernelPreferredCluster validates feasibility through the inherited can_implement path. That path evaluates cluster_shape_mn, which is set to fallback_cluster_shape_mn.

The preferred-cluster execution path also launches with preferred_cluster_shape_mn. The preferred shape currently has no equivalent feasibility validation before launch.

Required changes

In tensorrt_llm/_torch/cute_dsl_kernels/rubin/dense_bf16_gemm_persistent.py:

• Add or override feasibility validation for PersistentDenseGemmKernelPreferredCluster.
• Validate both fallback_cluster_shape_mn and preferred_cluster_shape_mn.
• Ensure __call__ rejects configurations that are invalid for either cluster shape before it launches a preferred or fallback grid.

Rationale

If the fallback cluster shape passes but the preferred cluster shape has more M CTAs than the problem supports, the preferred path can launch phantom CTAs. On SM107, this can cause out-of-bounds TMA accesses or a hang.

Acceptance criteria

• The preferred-cluster feasibility check evaluates both cluster shapes.
• The launch path uses the combined result.
• Tests cover a configuration where the fallback shape is feasible and the preferred shape is not.

Backlinks

• Pull request: https://github.com/NVIDIA/TensorRT-LLM/pull/18311
• Review comment: https://github.com/NVIDIA/TensorRT-LLM/pull/18311#discussion_r3872091284
• Requested by: @farazkh80