Audit fp32 random initialization for from-scratch pretraining

Description

Random weight initialization should sample from the intended distribution accurately. If tensors are initialized directly in a low-precision resident dtype such as bf16, the sampled distribution can be distorted before training starts. The safer pattern for from-scratch pretraining is to initialize in fp32, then cast back to the configured storage dtype.

We recently fixed this for DeepSeek-V3/Moonlight in https://github.com/NVIDIA-NeMo/Automodel/pull/2450 after observing unstable early pretraining when random weights were initialized directly in bf16 storage: very large early grad norms and flat loss. The same root issue can apply to any model or tensor that performs random initialization, although the visible harm may be strongest for MoE models because routing/expert dynamics are more sensitive to initialization quality.

User Impact

Most AutoModel users finetune from pretrained checkpoints, so they should not be affected. Users who train from config/random initialization with torch_dtype: bfloat16 may see poor convergence if a model initializes weights or other learned tensors directly in bf16.

Proposed Follow-Up

• Audit model families that support from_config / from-scratch pretraining.
• Identify all randomly initialized tensors, not only module weights.
• Ensure initialization samples in fp32, then casts back to the configured resident dtype.
• Add focused tests or smoke runs for high-risk configs, especially MoE models.

Priority

Medium. This is primarily a pretraining issue, not a known finetuning issue.

Start by locating the model families' from_config entry points and the code paths that randomly initialize weights or other learned tensors. Compare those paths with the DeepSeek-V3/Moonlight fix in PR #2450, then add focused tests or smoke runs for high-risk, especially MoE, configurations; done means random initialization samples in fp32 and returns tensors in the configured resident dtype.