mindspore-ai / mindspore-ai/hyper-parallel
[Feature]: 支持 DSA/MHC/MTP/shared expert 结构切分模板
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
🚀 功能描述
背景
distributed/_builder 的切分规划仅覆盖标准 attention/mlp/norm 边界,DSA/MLA 稀疏注意力、MTP 层、TP-extends-EP 下的 MoE shared expert 等模型专属结构只能由用户手写 plan override 才能正确切分;且 DSA 的 query/index 投影按 TP 切分后,缓存的 head 数挂在父 attention 模块上,现有逻辑只更新叶边界自身,导致 head 数不生效。
目标
- 为 DSA/MLA、MTP、MHC、shared expert 四类结构提供内置的切分模板,由 builder 按模块能力自动匹配,用户无需手写 override。
- 修正 TP 场景下 head 数的归属写入,并补充
num_index_headshead 属性。 loss_parallel_ops反传去除高级索引写入,改为掩码 + 安全索引的等价实现。
交付
- 新增模板:
dsa_template.py、mtp_template.py、mhc_template.py、shared_expert_template.py,注册进 planner 的结构模板 provider 列表。 - 规划接入:
recipe_spec.py新增 head-count 归属字段,rule_resolver.py纳入 override 合并,parameter_sharding.py在切分后写回归属模块。 - 配套用例:
tests/components/distributed/下四个模板用例、tests/ut/platform/torch/test_loss_parallel_ops.py。 - 规模:22 个文件,+2141 / -78。
验证
单卡 CPU 执行模板 UT 12 passed、loss parallel UT 6 passed;多卡 ST 与全量 UT 待补充。
关联 PR:<PR 链接>
现有替代方案
扩充现有的TP模板
与DFX相关性DF
提议的新API
是否影响现有API
无
补充信息
Thanks for contributing 🎉!
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 386
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/386
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing structure-template provider flow under distributed/_builder and read recipe_spec.py, rule_resolver.py, and parameter_sharding.py. Then inspect the four requested template tests under tests/components/distributed/ and test_loss_parallel_ops.py. Done means registering the four templates, handling head-count ownership and safe masked indexing, and passing the listed unit tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100