modelscope / modelscope/ms-swift
GRPO+LoRA训练过程中模型复读问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
我采用ms-swift中的LoRA+GRPO进行训练，发现当迭代几步后，模型的采样就会不断复读的情况了。下面是一个表现：
Your hardware and system info
我在之前的版本中没有遇到过这个问题，我测试了一下，貌似是从升级到3.9.2开始就有这个问题了，3.9.1没有该问题。不知道是不是vllm的版本问题还是什么原因。我安装环境的方式如下：
采用ms-swift版本对应的install_all.sh进行安装
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported GRPO+LoRA training run with ms-swift 3.9.1 and 3.9.2, using the installation method and environment described in the issue. Compare the installed vLLM versions and sampling behavior after several iterations. Done means identifying the version or dependency change associated with repeated output and confirming the behavior with a minimal reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100