[BUG]I found that the parameters of model will be fully transferred to the VRAM of each process. Is this abnormal in my understanding?

Describe the bug
I used the Transformers library with Deepspeed, and used Lora to fine-tune the CogVLM2 model. The parameter size of the model was 19B. During training, I used five graphics cards for training. I found that each graphics card loaded all the models during training. , resulting in the fact that even if the batch size is 1, an oom exception will still be reported. I checked the operating logic of Deepspeed.

I found that a detection will be performed here. When dont_change_device is false (this parameter defaults to False), or is_zero_init_model is False When the engine is initialized, the model will be transferred to the graphics card. The judgment logic of is_zero_init_model is ds_id. Now when processing the Engine initialization stage, ds_id will definitely not exist, so is_zero_init_model must be False, so the model will be fully transferred into the graphics memory. This Is there any conflict with Deepspeed's idea of ​​reducing video memory usage? I hope the developer can answer my questions. Thank you very much.

Log output

To Reproduce
ds_config.json

  "bf16": {
    "enabled": true
  },
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": "auto",
      "betas": "auto",
      "eps": "auto",
      "weight_decay": "auto"
    }
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto",
      "warmup_num_steps": "auto"
    }
  },
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 2e8,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 2e8,
    "stage3_max_reuse_distance": 2e8,
    "stage3_gather_16bit_weights_on_model_save": true
  },
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "steps_per_print": 1000,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "wall_clock_breakdown": false
}

trainingArguments = {
    "output_dir": output_dir_new,
    "num_train_epochs": 2,
    "per_device_train_batch_size": 1,
    "per_device_eval_batch_size":1,
    "warmup_steps": 100,
    "weight_decay": 0.01,
    "logging_dir": "./logs",
    "logging_steps": 10,
    "do_train": True,
    "do_eval": True,
    "do_predict": True,
    "predict_with_generate": True,
    "save_strategy": "epoch",
    "save_steps": 1000,
    "generation_config": None,
    "dataloader_pin_memory": False,
    # "bf16": False,
    "bf16": True,
    deepspeed":"ds_config_bf16.json"
    }

Expected behavior
I expect the model to load normally using Deepspeed

ds_report output
Please run ds_report to give us details about your setup.

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

Docker context
Are you using a specific docker image that you can share?

Additional context
Add any other context about the problem here.
The CogVLM Model info