mindspore-ai / mindspore-ai/hyper-parallel

[Feature]: 支持 DSA/MHC/MTP/shared expert 结构切分模板

Open
#148 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

🚀 功能描述
背景

distributed/_builder 的切分规划仅覆盖标准 attention/mlp/norm 边界,DSA/MLA 稀疏注意力、MTP 层、TP-extends-EP 下的 MoE shared expert 等模型专属结构只能由用户手写 plan override 才能正确切分;且 DSA 的 query/index 投影按 TP 切分后,缓存的 head 数挂在父 attention 模块上,现有逻辑只更新叶边界自身,导致 head 数不生效。

目标
  1. 为 DSA/MLA、MTP、MHC、shared expert 四类结构提供内置的切分模板,由 builder 按模块能力自动匹配,用户无需手写 override。
  2. 修正 TP 场景下 head 数的归属写入,并补充 num_index_heads head 属性。
  3. loss_parallel_ops 反传去除高级索引写入,改为掩码 + 安全索引的等价实现。
交付
  • 新增模板:dsa_template.pymtp_template.pymhc_template.pyshared_expert_template.py,注册进 planner 的结构模板 provider 列表。
  • 规划接入:recipe_spec.py 新增 head-count 归属字段,rule_resolver.py 纳入 override 合并,parameter_sharding.py 在切分后写回归属模块。
  • 配套用例:tests/components/distributed/ 下四个模板用例、tests/ut/platform/torch/test_loss_parallel_ops.py
  • 规模:22 个文件,+2141 / -78。
验证

单卡 CPU 执行模板 UT 12 passed、loss parallel UT 6 passed;多卡 ST 与全量 UT 待补充。

关联 PR:<PR 链接>

现有替代方案

扩充现有的TP模板

与DFX相关性DF
提议的新API
是否影响现有API

补充信息

Thanks for contributing 🎉!

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 386
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/386

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing structure-template provider flow under distributed/_builder and read recipe_spec.py, rule_resolver.py, and parameter_sharding.py. Then inspect the four requested template tests under tests/components/distributed/ and test_loss_parallel_ops.py. Done means registering the four templates, handling head-count ownership and safe masked indexing, and passing the listed unit tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.