modelscope / modelscope/ms-swift
[BUG] 4.0.0版本使用PiSSA进行权重初始化导致eval_loss值异常
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述
ms-swift4.0.0版本使用PiSSA进行权重初始化然后进行SFT,eval_loss值异常,具体:
-
Loss:train_loss正常下降,eval_loss一直为一个固定值(例如首次验证为eval_loss=0.016,后续验证时eval_loss始终是0.016);
-
checkpoints:保存的各个checkpoints都能够正常部署,且模型表现正常(符合微调之后的表现)。
How to Reproduce / 如何复现
ms-swift版本:4.0.0
transformers:5.2.0
peft:0.18.1
使用如下脚本eval_loss不会变化(一直是一个固定值,并不会随着训练进行而改变):
CUDA_VISIBLE_DEVICES=0
swift sft
--model Qwen/Qwen3-4B
--template qwen3_nothinking
--model_type qwen3
--train_type lora
--init_weights pissa
--dataset 'swift/Qwen3-SFT-Mixin#2000'
--load_from_cache_file true
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--learning_rate 1e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 16
--eval_steps 50
--save_steps 50
--save_total_limit 2
--logging_steps 5
--max_length 2048
--output_dir output
--warmup_ratio 0.05
--dataloader_num_workers 4
--model_author swift
--model_name swift-robot
将init_weights改为true后eval_loss正常:
CUDA_VISIBLE_DEVICES=0
swift sft
--model Qwen/Qwen3-4B
--template qwen3_nothinking
--model_type qwen3
--train_type lora
--init_weights true
--dataset 'swift/Qwen3-SFT-Mixin#2000'
--load_from_cache_file true
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--learning_rate 1e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 16
--eval_steps 50
--save_steps 50
--save_total_limit 2
--logging_steps 5
--max_length 2048
--output_dir output
--warmup_ratio 0.05
--dataloader_num_workers 4
--model_author swift
--model_name swift-robot
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue through the swift sft entry point using the provided Qwen3 command and the listed ms-swift, transformers, and peft versions. Compare PiSSA with init_weights true, then trace the SFT evaluation path and PiSSA integration. Done means eval_loss changes across evaluation steps while checkpoints retain the reported behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100