Dont_change_device for parameters in initialization

When I was running model training by Zero offload, to save the GPU memory I make the model weights initialized on CPU memory too by setting up deepspeed.zero.Init(remote_device="cpu", dtype=torch.half, enabled=False). Although the model weight is really initialized on CPU memory, but after deepspeed.initialzed(), the model still move to GPU memory. So I am wondering

1. in Zero offload (stage 3, offload to cpu/nvme) is it possible the weight of the model stay mainly on CPU memory/Nvme but only is loaded layer by layer to GPU memory?
2. I found in engine.py (actually it is called by deepspeed.initialize()) there is an argument dont_change_device (link)[https://github.com/microsoft/DeepSpeed/blob/4ae3a3da0dfd19d7ab7a76e7c742ac12f44fc1c0/deepspeed/runtime/engine.py#L1138-L1139] which controls whether or not the model weights is moved to GPU memory. But I also found no place to call dont_change_device. So my question is how to use dont_change_device and is it used to retain the model weight on CPU memory?

Start with the dont_change_device argument in deepspeed/runtime/engine.py around lines 1138-1139 and trace how deepspeed.initialize() handles the model after zero.Init(remote_device="cpu", dtype=torch.half, enabled=False). Check whether the argument is reachable and whether Zero stage 3 offload preserves model weights on CPU or NVMe; done means the behavior and supported usage are documented or covered by a test.