`_cast_module_mixed_precision` casts every dtype torch calls floating point, corrupting FP8/MX/NVFP4 parameters

Repo: deepspeedai/DeepSpeed
Version: DeepSpeed 0.19.2, PyTorch 2.11.0+cu130

Problem

DeepSpeedEngine._cast_module_mixed_precision (deepspeed/runtime/engine.py:1608) casts every
parameter for which p.is_floating_point() is true:

for p in self.module.parameters(recurse=True):
    if p.is_floating_point() and p.dtype != param_dtype:
        p.data = p.data.to(param_dtype)

is_floating_point() is true for the narrow-precision storage dtypes as well, so quantized
parameters that are frozen, never receive gradients, and are consumed directly by a
quantization-aware GEMM are swept into the mixed-precision cast. On PyTorch 2.11:

Three distinct failure modes:

• FP8 weights double from one byte per element to two.
• MX scales (float8_e8m0fnu) hold a power-of-two exponent, not a value. Casting to bf16 and
  back does not round-trip the encoding.
• NVFP4 (float4_e2m1fn_x2) packs two 4-bit values per byte and has no copy_ implementation,
  so deepspeed.initialize fails outright on any model holding such a parameter.

This matters for the increasingly common setup of training LoRA adapters on top of a checkpoint
held in its native quantized form.

Reproduction

import torch, deepspeed, torch.nn as nn

DTYPE = torch.float8_e4m3fn  # or float8_e8m0fnu, or float4_e2m1fn_x2 to see the crash

class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.frozen = nn.Parameter(torch.zeros(4096, 4096, dtype=DTYPE), requires_grad=False)
        self.adapter = nn.Linear(4096, 16, bias=False)  # the only trainable tensor

m = M()
print("before:", m.frozen.dtype, m.frozen.numel() * m.frozen.element_size() // 2**20, "MiB")

engine, *_ = deepspeed.initialize(
    model=m,
    model_parameters=[p for p in m.parameters() if p.requires_grad],
    config={
        "train_batch_size": 1,
        "bf16": {"enabled": True},
        "zero_optimization": {"stage": 2},
    },
)
p = engine.module.frozen
print("after: ", p.dtype, p.numel() * p.element_size() // 2**20, "MiB")

Observed with float8_e4m3fn: 16 MiB before, bfloat16, 32 MiB after.
Observed with float4_e2m1fn_x2: NotImplementedError during initialize.
Expected in both cases: the parameter is left untouched.