zero stage 3 cpu offload consumes less cpu memory than zero stage 2 cpu offload ?

Hi, I want to train a model with 100B model params using 8 nodes and each node has 8 nvidia V100 gpu. Each node has 376GB cpu memory. I did my experiments using the following settings:
setting1：zero stage 2 & cpu offload， mp=16， dp=4
setting2：zero stage 3 & cpu offload，mp=8，dp=8

setting 1 didn't work，because cpu memory is not enough, process was killed due to OOM, However setting 2 did work, consumed about 220G cpu memory. My zero 3 settings is :
"zero_optimization": {
"stage": 3,
"cpu_offload": true,
"cpu_offload_params": true,
"overlap_comm": true,
"contiguous_gradients": true,
"stage3_max_live_parameters": 6000000,
"stage3_max_reuse_distance": 100000000,
"stage3_prefetch_bucket_size": 200000,
"stage3_param_persitance_threshold": 100000,
"reduce_bucket_size": 3000000,
"sub_group_size": 1e6
}

According to my previous knowledge, stage3 cpu offload occupies more memory than stage 2, but experiments show that this is not the case. Is there something wrong with me?

The issue names no files, tests, or entry points. Start by comparing the reported ZeRO stage 2 and stage 3 CPU-offload configurations and their parallelism settings, then trace how CPU memory is accounted for in each mode. Done means explaining the observed memory difference or identifying a reproducible configuration problem.