modelscope / modelscope/ms-swift

lora 微调的模型使用--resume_from_checkpoint参数,继续训练报显存不足;不使用--resume_from_checkpoint参数可以正常训练

Open
#2,505 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程,最好有截图)
lora微调脚本
如下是lora微调脚本,如果使用 --resume_from_checkpoint 加载微调后的模型继续训练,就会报错。显卡型号:A800-40G显存
如果不使用 --resume_from_checkpoint 参数微调模型,是可以训练的。

nproc_per_node=8
max_length=8300
model_id_or_path=./Qwen2-7B-Instruct
model_type=qwen2-7b-instruct

NPROC_PER_NODE=$nproc_per_node \
MASTER_PORT=29500 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
swift sft \
    --model_id_or_path $model_id_or_path \
    --model_type $model_type \
    --model_revision master \
    --sft_type lora \
    --tuner_backend peft \
    --template_type AUTO \
    --dtype bf16 \
    --output_dir output \
    --ddp_backend nccl \
    --custom_train_dataset_path train.json \
    --lora_rank 16 \
    --lora_alpha 64 \
    --lora_dropout_p 0.05 \
    --lora_target_modules DEFAULT \
    --custom_val_dataset_path test_2.json \
    --train_dataset_sample -1 \
    --num_train_epochs 1 \
    --max_length $max_length \
    --check_dataset_strategy warning \
    --gradient_checkpointing true \
    --batch_size 1 \
    --weight_decay 0.1 \
    --learning_rate 3e-4 \
    --gradient_accumulation_steps $(expr 128 / $nproc_per_node) \
    --max_grad_norm 0.5 \
    --warmup_ratio 0.03 \
    --eval_steps 1000 \
    --save_steps 1000 \
    --save_total_limit 10 \
    --logging_steps 10 \
    --use_flash_attn true \
    --save_only_model true \
    --lazy_tokenize true \
    --deepspeed 'default-zero2' \
    --resume_from_checkpoint output/qwen2-7b-instruct/v14-20241123-110642/checkpoint-8790 \
    --resume_only_model true \

错误截图
image

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息,如CUDA版本,系统,GPU型号和torch版本等)
torch 2.1.2+cu121
ms-swift 2.6.0.post2
transformers 4.42.0
NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.1

Additional context
Add any other context about the problem here(在这里补充其他信息)
在A800-80G的显卡的机器上,可以使用 --resume_from_checkpoint参数来微调模型。只是占用显存比不使用 --resume_from_checkpoint参数的要大一些。

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the swift sft entry point and reproduce the provided multi-GPU command with and without --resume_from_checkpoint, comparing the 40G and 80G results. Trace checkpoint-resume handling and memory usage, then verify that resumed LoRA training fits the documented hardware and preserves the expected training behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.