deepspeedai / deepspeedai/DeepSpeed
[BUG] AttributeError: 'Linear' object has no attribute 'ds_grads_remaining'
Open
Nobody has claimed this yet.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
When I use llamafactory to lora sft on qwen2.5vl-7b with ds-z3, I met:
Traceback (most recent call last):
File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/bin/torchrun", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/run.py", line 918, in main
run(args)
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1845, in _call_impl
[rank1]: return inner()
[rank1]: ^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1793, in inner
[rank1]: result = forward_call(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/peft/tuners/tuners_utils.py", line 193, in forward
[rank1]: return self.model.forward(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1875, in forward
[rank1]: logits = self.lm_head(hidden_states)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1845, in _call_impl
[rank1]: return inner()
[rank1]: ^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1782, in inner
[rank1]: args_result = hook(self, args)
[rank1]: ^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 372, in _post_backward_module_hook
[rank1]: return apply_to_tensors_only(module.post_bwd_fn.apply,
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/utils.py", line 133, in apply_to_tensors_only
[rank1]: touched_output = apply_to_tensors_only(function, elem)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/utils.py", line 149, in apply_to_tensors_only
[rank1]: touched_output = function(value)
[rank1]: ^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/autograd/function.py", line 575, in apply
[rank1]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 440, in forward
[rank1]: module.ds_grads_remaining += 1
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__
[rank1]: raise AttributeError(
[rank1]: AttributeError: 'Linear' object has no attribute 'ds_grads_remaining'
My ds_z3_config:
{
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"zero_allow_untested_optimizer": true,
"fp16": {
"enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"bf16": {
"enabled": "auto"
},
"zero_optimization": {
"stage": 3,
"overlap_comm": false,
"contiguous_gradients": true,
"sub_group_size": 1e9,
"reduce_bucket_size": "auto",
"stage3_prefetch_bucket_size": "auto",
"stage3_param_persistence_threshold": "auto",
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_gather_16bit_weights_on_model_save": true
}
}
To Reproduce
Steps to reproduce the behavior:
llamafactory-cli train train.yaml
my train.yaml:
### model
model_name_or_path: "{{BASE_MODEL_SUBDIR}}/{{BASE_MODEL_NAME}}"
trust_remote_code: true
new_special_tokens: "{{NEW_SPECIAL_TOKENS}}"
skip_special_tokens: false
torch_empty_cache_steps: 1
image_max_pixels: 15680
video_max_pixels: 31360
video_fps: 2.0
video_maxlen: 48
### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_alpha: 16
lora_target: all
lora_dropout: 0.05
deepspeed: "{{BASE_PATH}}/LLaMA-Factory/examples/deepspeed/ds_z3_config.json" # ds_z3_config # ds_z3_offload_config
# enable_liger_kernel: True
# use_unsloth_gc: True
### dataset
dataset_dir: "{{BASE_PATH}}/LLaMA-Factory/data/"
dataset: "CPED_RE_{{TASK}}" # video_annotation, audio_chat, audio_annotation
template: qwen2_vl
cutoff_len: 2048 # 14800 # 2048 # 16384
max_samples: 320 # for DEBUG
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4
### output
output_dir: "{{BASE_PATH}}/models/checkpoints/{{BASE_MODEL_NAME}}_train_lora_{{TASK}}_{{CURRENT_DATE}}"
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 4
learning_rate: 1.0e-5
num_train_epochs: 2.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
# resume_from_checkpoint: null
### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500
# swanlab
use_swanlab: true
swanlab_project: llamafactory
swanlab_run_name: "{{BASE_MODEL_NAME}}_train_lora_{{TASK}}_{{CURRENT_DATE}}"
Note that I use sed s to replace the placeholders above in bash temporarily and that's all.
ds_report output
[2025-04-04 02:29:57,350] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
[WARNING] FP Quantizer is using an untested triton version (3.2.0), only 2.3.(0, 1) and 3.0.0 are known to be compatible with these kernels
fp_quantizer ........... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
/home/mas-wang.zhenyu/anaconda3/envs/vllm/compiler_compat/ld: cannot find -lcufile: No such file or directory
collect2: error: ld returned 1 exit status
gds .................... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.6
[WARNING] using untested triton version (3.2.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch']
torch version .................... 2.6.0+cu118
deepspeed install path ........... ['/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed']
deepspeed info ................... 0.16.4, unknown, unknown
torch cuda version ............... 11.8
torch hip version ................ None
nvcc version ..................... 11.6
deepspeed wheel compiled w. ...... torch 2.6, cuda 11.8
shared memory (/dev/shm) size .... 110.01 GB
System info (please complete the following information):
[2025-04-03 06:02:20,328] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO 04-03 06:02:21 [init.py:256] Automatically detected platform cuda.
- llamafactory version: 0.9.3.dev0
- Platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.31
- Python version: 3.12.9
- PyTorch version: 2.6.0+cu118 (GPU)
- Transformers version: 4.50.0
- Datasets version: 3.4.1
- Accelerate version: 1.5.2
- PEFT version: 0.15.0
- TRL version: 0.9.6
- GPU type: NVIDIA GeForce RTX 4090
- GPU number: 8
- GPU memory: 23.65GB
- DeepSpeed version: 0.16.5
- vLLM version: 0.8.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce with llamafactory-cli train train.yaml using the supplied Qwen2.5-VL LoRA and ZeRO-3 configuration. Start by tracing the failure through deepspeed/runtime/zero/parameter_offload.py and compare the involved module setup with the reported PyTorch 2.6 and DeepSpeed environment. Done means the same training configuration completes without the ds_grads_remaining AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100