deepspeedai / deepspeedai/DeepSpeed

Deepspeed zero 3 seems to save empty adapter_model.safetensors

Open
#7,900 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

I am using llamafactory with deepspeed stage3 + QLoRA. I trained with SFT and stored checkpoints. The checkpoints had

adapter_config.json        chat_template.jinja  special_tokens_map.json  trainer_state.json     zero_to_fp32.py
adapter_model.safetensors  latest               tokenizer.json           training_args.bin global_step

I assumed that the adapter_model.safetensors were adapter weights but they seem to be empty.
Can someone please clarify what gets stored in the global_step folder and how can i save the adapter_weights only and not save the base model while checkpointing?

I am using 4 A100 and training qwen3-235B0thinking and used cpu offloading

  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "zero_allow_untested_optimizer": true,
  "bf16": {
    "enabled": "auto"
  },
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": true
    },
    "overlap_comm": true,
    "contiguous_gradients": true,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 5e8,
    "stage3_max_reuse_distance": 5e8,
    "stage3_gather_16bit_weights_on_model_save": false
  }
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the checkpoint with the provided DeepSpeed stage 3 configuration, then inspect adapter_model.safetensors and the global_step folder. Compare the stored artifacts with the requested adapter-only checkpoint behavior; done means clarifying what each artifact contains and documenting or correcting the save behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.