[BUG] Trainer saves global_steps300 in LoRA training with deepspeed

Describe the bug
I trained LLama 2 with deepspeed/
Trainer with 2 GPU but on saving the checkpoint with the following configuration deepspeed saves a large folder global_step50, which is 44GB. How I can automatically not save this folder? I just need adapter checkpoints.

To Reproduce
Steps to reproduce the behavior:

{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "bf16": {
        "enabled": "auto"
    },

    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "none",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "contiguous_gradients": true
    },

    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "steps_per_print": 100,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}

Expected behavior
For LoRA training, I just need adapter as checkpoints.

Screenshots

Start by reproducing the LoRA training run with the provided DeepSpeed configuration and inspect how checkpoints are written. Done means the training produces adapter checkpoints without the large global_step folder, while preserving the expected checkpoint behavior.