modelscope / modelscope/ms-swift
qwen3.5-4b模型对于padding_free的支持问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述
在ms-swift源码中:
swift 的 sp_linear_forward 内部(简化示意)
def sp_linear_forward(self, hidden_states, **kwargs):
if not sequence_parallel.enabled() and 'cu_seq_lens_q' not in kwargs:
# 第一分支:走 transformers 原始 forward(fla CUDA 内核,正常)
return original_forward(self, hidden_states, **kwargs)
else:
# 第二分支:走 swift varlen 路径 → 尝试 import flash_linear_attention → 崩溃!
return _run_sequence_parallel_forward(self, hidden_states, **kwargs)
此处的条件判断由于伪packing机制导致所有训练无论batch_size的大小都会走varlen实现去跑,但是varlen中实际没有写回退原生torch实现的方法。而Windows中目前虽然有fla和triton实现,但是没有合现成的causal-conv1d whl,因此出现ExceptImport。
How to Reproduce / 如何复现
1
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the described sp_linear_forward condition and the _run_sequence_parallel_forward path in the ms-swift source, comparing them with original_forward for qwen3.5-4b and padding_free training. Reproduce the Windows failure involving flash_linear_attention and the missing causal-conv1d wheel; done means training no longer crashes when the varlen dependencies are unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100