much more memory used than In theory

I am training glm-10b-chinese for step-1.
In theory, 10b paramters, fp32, total memory occupied should be :

• params : 40GB
• gradian : 40GB
• optim : 40GB * 3 = 120GB
  200GB totaly.

even load all params and optimizer params into CPU memory, add some buffers, 300GB CPU memory is sufficient. However, CPU memory occupied actually is almost 450GB（peak value）:

• GPU momey : 51.5 GB
• CPU memory : 445GB （peak value）, 21GB (The value after stabilization)

My question is : why use too much cpu memory, something wrong with my Settings?

Settings:
zero-stage : zero-2
node : 2 or 4
gpu : 8 * A100 * 80G , per node

deepspeed steps/train/train_sft.py \ --run_name ${RUN_NAME} \ --train_filename $TRAIN_FILENAME \ --val_filename $VAL_FILENAME \ --eval_interval 64 \ --checkpoint_save_interval 512 \ --model_type "THUDM/glm-10b-chinese" \ --model_name_or_path "THUDM/glm-10b-chinese" \ --per_device_train_batch_size 1 \ --per_device_val_batch_size 1 \ --max_prompt_length 512 \ --max_gen_length 512 \ --learning_rate 5e-5 \ --frozen_ratio 0.3 \ --weight_decay 0.1 \ --num_train_epochs 1 \ --lr_scheduler_type cosine \ --gradient_accumulation_steps 10 \ --gradient_checkpointing \ --disable_dropout \ --num_warmup_steps 100 \ --deepspeed --seed 42 \ --offload \ --zero_stage 2 \ --output_dir $OUTPUT \

Start with steps/train/train_sft.py and the supplied command, focusing on the zero-stage 2 and CPU offload settings. Reproduce the run with the linked glm-10b-chinese configuration and compare the reported GPU and peak CPU memory. Done means identifying whether the memory peak is expected or caused by a setting, with evidence and a documented correction.