modelscope / modelscope/ms-swift

[BUG] 4.0.0版本使用PiSSA进行权重初始化导致eval_loss值异常

Open
#8,271 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述

ms-swift4.0.0版本使用PiSSA进行权重初始化然后进行SFT,eval_loss值异常,具体:

  • Loss:train_loss正常下降,eval_loss一直为一个固定值(例如首次验证为eval_loss=0.016,后续验证时eval_loss始终是0.016);

  • checkpoints:保存的各个checkpoints都能够正常部署,且模型表现正常(符合微调之后的表现)。

How to Reproduce / 如何复现

ms-swift版本:4.0.0
transformers:5.2.0
peft:0.18.1

使用如下脚本eval_loss不会变化(一直是一个固定值,并不会随着训练进行而改变):

CUDA_VISIBLE_DEVICES=0
swift sft
--model Qwen/Qwen3-4B
--template qwen3_nothinking
--model_type qwen3
--train_type lora
--init_weights pissa
--dataset 'swift/Qwen3-SFT-Mixin#2000'
--load_from_cache_file true
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--learning_rate 1e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 16
--eval_steps 50
--save_steps 50
--save_total_limit 2
--logging_steps 5
--max_length 2048
--output_dir output
--warmup_ratio 0.05
--dataloader_num_workers 4
--model_author swift
--model_name swift-robot

将init_weights改为true后eval_loss正常:

CUDA_VISIBLE_DEVICES=0
swift sft
--model Qwen/Qwen3-4B
--template qwen3_nothinking
--model_type qwen3
--train_type lora
--init_weights true
--dataset 'swift/Qwen3-SFT-Mixin#2000'
--load_from_cache_file true
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--learning_rate 1e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 16
--eval_steps 50
--save_steps 50
--save_total_limit 2
--logging_steps 5
--max_length 2048
--output_dir output
--warmup_ratio 0.05
--dataloader_num_workers 4
--model_author swift
--model_name swift-robot

Additional Information / 补充信息

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue through the swift sft entry point using the provided Qwen3 command and the listed ms-swift, transformers, and peft versions. Compare PiSSA with init_weights true, then trace the SFT evaluation path and PiSSA integration. Done means eval_loss changes across evaluation steps while checkpoints retain the reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.