mindspore-ai / mindspore-ai/hyper-parallel
[Bug]: dualmode Trainer FSDP optimizer layout loses TP global shape
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
Checklist
- 已检索现有 Issue。
- 已阅读相关设计和实现。
- 已通过 dualmode Trainer 多维并行用例复现。
问题描述
dualmode Trainer 生产路径中,TP/EP 处理后的参数在进入 fully_shard 时是普通本地 Parameter,不再携带 DTensor 的 global shape。FSDP 参数管理若直接把传入参数的 local shape 当作 optimizer Layout 的 global shape,会造成元数据与 FSDP+TP/EP placements 不匹配。
Muon 根据错误的 optimizer layout 恢复 Newton-Schulz 输入形状时,会在 TP+FSDP 场景触发 shape is invalid for input of size 16384。多维 replicate mesh 下,异步 relay broadcast 若未等待前一维通信完成,还可能让后一维从尚未收到有效数据的 rank 继续广播,污染 optimizer 参数。
复现场景:8 卡 HSDP(2×2)+TP(2)、DP(2)+CP(2)+TP(2) 以及 HSDP+TP+EP MoE dualmode Trainer,用优化器执行参数更新并与 standalone 模型逐步对拍。
期望行为
FSDP 应根据 TPShardMetaInfo 中已知的 mesh 和 placements,从 TP/EP-local shape 恢复 logical global shape,再构造统一的 FSDP+TP/EP optimizer layout。Muon 应同时支持普通 Parameter 与 DTensor,并保证多维 replicate relay 的维度间数据依赖。
环境信息
- 平台:PyTorch,
HYPER_PARALLEL_PLATFORM=torch - 硬件:8×Ascend 910B
- 目标分支:
trainer_dev - 并行组合:HSDP+TP、DP+CP+TP、HSDP+TP+EP MoE
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 330
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/330
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing dualmode Trainer parameter handling into FSDP fully_shard, focusing on TPShardMetaInfo, optimizer layout construction, Muon shape restoration, and multidimensional replicate relay. Reproduce the listed HSDP+TP or DP+CP+TP cases, then verify that optimizer updates match the standalone model step by step and that the MoE HSDP+TP+EP path preserves relay dependencies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100