[Question]Significant differences between deepspeed and torchrun training results

Hello,

I'm performing continual training of Llama-2-7b. I'm using the same hyperparameters to train the model (with flashattn2 and bf16 on A100) using torchrun and deepspeed (zero2) but the results are very different.

Torchrun's results are significantly better than deepspeed trained ones:

My deepspeed configuration is:

{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "bf16": {
           "enabled": "auto"
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupDecayLR",
        "params": {
            "warmup_min_lr": 5e-6,
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto",
            "warmup_type": "linear",
            "total_num_steps": "auto"
        }
    },

    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "contiguous_gradients": true
    },

    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"
}

I checked the differences between torchrun and deepspeed. The only possible difference is on the warmup_min_lr (torchrun using 0 but deepspeed using 5e-6) and optimizer (torchrun using adamw_torch, deepspeed using its native AdamW). Except these differences, I can't find any other possible reasons accounting for the difference of the performance. However, I don't think these two configuration differences could lead to such a significant performance gap. Is there anything that I missed? Any suggestion or comment would be very helpful to me.

Thank you

Start by reproducing the Llama-2-7b continual-training comparison on A100 hardware with the posted DeepSpeed configuration and the torchrun setup. Compare the warmup_min_lr, optimizer, and resulting training curves; done means identifying a supported explanation for the discrepancy or documenting the missing configuration difference.