modelscope / modelscope/ms-swift

qwen3.5-4b模型对于padding_free的支持问题

Open
#9,845 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述

在ms-swift源码中:

swift 的 sp_linear_forward 内部(简化示意)

def sp_linear_forward(self, hidden_states, **kwargs):
if not sequence_parallel.enabled() and 'cu_seq_lens_q' not in kwargs:
# 第一分支:走 transformers 原始 forward(fla CUDA 内核,正常)
return original_forward(self, hidden_states, **kwargs)
else:
# 第二分支:走 swift varlen 路径 → 尝试 import flash_linear_attention → 崩溃!
return _run_sequence_parallel_forward(self, hidden_states, **kwargs)

此处的条件判断由于伪packing机制导致所有训练无论batch_size的大小都会走varlen实现去跑,但是varlen中实际没有写回退原生torch实现的方法。而Windows中目前虽然有fla和triton实现,但是没有合现成的causal-conv1d whl,因此出现ExceptImport。

How to Reproduce / 如何复现

1

Additional Information / 补充信息

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the described sp_linear_forward condition and the _run_sequence_parallel_forward path in the ms-swift source, comparing them with original_forward for qwen3.5-4b and padding_free training. Reproduce the Windows failure involving flash_linear_attention and the missing causal-conv1d wheel; done means training no longer crashes when the varlen dependencies are unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.