[BUG] Some problems about zero++ while training Llama2-70B

In the past few days, I have tried to conduct pre-training experiments with llama2-7b and llama2-70b using zero++. I encountered some problems:

1. When using zero++ for multi-machine and multi-card training, the number of GPUs on each of my nodes is 8. I set the zero_hpz_partition_size to 8 and found that it can run normally with 1 * 8, 2 * 8, and 4 * 8 cards. However, in the case of 3*8 cards, there is an error:

AssertError: Quantized weight requires the number of weights to be a multiple of 8

If I modify the padding_size to be a multiple of 8, then there is another error:

RuntimeError: output tensor size must be equal to world_size times input tensor size issue.

2. I used the code from this GitHub:(https://github.com/hiyouga/LLaMA-Efficient-Tuning) to train llama2-70b. I found that with 3*8 cards, batch size=1, length=2048, fp16 and accumulate=4, the speed of zero++ did not improve and was even slower than zero3: zero3 vs zero++ : 29s/iter vs 33s/iter. The following is my zero++ config:

{
  "fp16": {
      "enabled": "auto",
      "loss_scale": 0,
      "loss_scale_window": 1000,
      "initial_scale_power": 16,
      "hysteresis": 2,
      "min_loss_scale": 1,
      "fp16_opt_level": "O2"
  },

  "bf16": {
      "enabled": "auto"
  }, 

  "optimizer": {
      "type": "AdamW",
      "params": {
          "lr": "auto",
          "betas": "auto",
          "eps": "auto",
          "weight_decay": "auto"
      }
  },

  "scheduler": {
      "type": "WarmupDecayLR",
      "params": {
          "last_batch_iteration": -1,
          "total_num_steps": "auto",
          "warmup_min_lr": "auto",
          "warmup_max_lr": "auto",
          "warmup_num_steps": "auto"
      }
  },

  "zero_optimization": {
      "stage": 3,
      "overlap_comm": true,
      "contiguous_gradients": true,
      "sub_group_size": 1e9,
      "reduce_bucket_size": "auto",
      "stage3_prefetch_bucket_size": "auto",
      "stage3_param_persistence_threshold": "auto",
      "stage3_max_live_parameters": 1e9,
      "stage3_max_reuse_distance": 1e9,
      "gather_16bit_weights_on_model_save": true,
      
      "zero_quantized_weights": true,
      "zero_hpz_partition_size": 8,
      "zero_quantized_gradients": true
  },

  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "steps_per_print": 2000,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "wall_clock_breakdown": false
}

3. I noticed an issue with unstable loss during training, which occurs in both 7b and 70b. The loss would increase during the training process. When training with 7b, I also encountered the error

Exception: Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.

I tried setting zero_quantized_gradients to false, but the problem was not resolved.

I'm not sure if I did something wrong in my operations. I would like to ask if there are solutions to these problems, thanks.

The issue names no source files, tests, or entry points; begin by reproducing the posted zero++ configuration on 3×8 GPUs and compare it with ZeRO-3. Investigate the quantization assertion, tensor-size error, performance difference, and loss-scale failures separately; done means each behavior has a confirmed cause and a documented fix or reproduction boundary.