Temporarily disable TE fused RoPE globally (breaks HF/vLLM logprob parity in some models)

Summary

TE's fused RoPE kernel is being temporarily disabled globally across all model families. The fused kernel computes cos/sin in fp32 in-kernel, while HF and vLLM rotate with bf16 cos/sin tables. This mismatch breaks logprob parity against HF reference models and against vLLM rollouts (RL), and is not confined to a single family.

Why / impact

• Confirmed to cause divergence in Qwen3-MoE (logprob divergence vs vLLM; see related items below).
• The same fp32-vs-bf16 RoPE discrepancy is a latent risk for every family that can use the fused path — so we are turning it off everywhere rather than family-by-family, until a proper parity-preserving fix lands.
• Trade-off: fused RoPE is a throughput optimization; disabling it will slightly increase attention-block cost. This is an intentional, temporary correctness-over-speed choice.

What changed

Force rope_fusion = False in BackendConfig.__post_init__ (nemo_automodel/components/models/common/utils.py) — the single chokepoint every BackendConfig passes through (including the dataclasses.replace() in resolve_rope_fusion). This overrides:

1. the legacy None -> (HAVE_TE and cuda) -> True default,
2. any explicit rope_fusion: true set in a recipe/config,
3. both the resolver-based families and the families that read backend.rope_fusion directly (llama, qwen2, deepseek_v3, kimi_k25_vl, kimivl, mistral4).

A one-time rank-0 warning is logged when an explicit True is overridden.

Scope

• All model families, all recipes, training + inference/rollout paths.
• No config changes required from users; the toggle is code-level and centralized.

Exit criteria (to re-enable)

• Root-cause fix so the fused RoPE kernel is numerically consistent with bf16 HF/vLLM RoPE (or the reference paths are aligned to the fused kernel), verified by logprob-parity tests against HF and vLLM.
• Remove the override block in BackendConfig.__post_init__ to restore per-family behavior.

Related

• Qwen3-MoE vLLM logprob divergence (root cause: TE fused RoPE fp32 vs bf16 cos/sin).
• AM-656 (KL divergence between original and HF-loaded model).

Start in nemo_automodel/components/models/common/utils.py at BackendConfig.post_init, and inspect merged pull request #3028 before making changes. Verify the global rope_fusion behavior and the logprob-parity exit criteria against HF and vLLM; the work is done when the fused path is disabled consistently until a parity-preserving fix is available.