mindspore-ai / mindspore-ai/hyper-parallel

[Bug]: dualmode Trainer FSDP optimizer layout loses TP global shape

Open
#189 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

Checklist
  • 已检索现有 Issue。
  • 已阅读相关设计和实现。
  • 已通过 dualmode Trainer 多维并行用例复现。
问题描述

dualmode Trainer 生产路径中,TP/EP 处理后的参数在进入 fully_shard 时是普通本地 Parameter,不再携带 DTensor 的 global shape。FSDP 参数管理若直接把传入参数的 local shape 当作 optimizer Layout 的 global shape,会造成元数据与 FSDP+TP/EP placements 不匹配。

Muon 根据错误的 optimizer layout 恢复 Newton-Schulz 输入形状时,会在 TP+FSDP 场景触发 shape is invalid for input of size 16384。多维 replicate mesh 下,异步 relay broadcast 若未等待前一维通信完成,还可能让后一维从尚未收到有效数据的 rank 继续广播,污染 optimizer 参数。

复现场景:8 卡 HSDP(2×2)+TP(2)、DP(2)+CP(2)+TP(2) 以及 HSDP+TP+EP MoE dualmode Trainer,用优化器执行参数更新并与 standalone 模型逐步对拍。

期望行为

FSDP 应根据 TPShardMetaInfo 中已知的 mesh 和 placements,从 TP/EP-local shape 恢复 logical global shape,再构造统一的 FSDP+TP/EP optimizer layout。Muon 应同时支持普通 Parameter 与 DTensor,并保证多维 replicate relay 的维度间数据依赖。

环境信息
  • 平台:PyTorch,HYPER_PARALLEL_PLATFORM=torch
  • 硬件:8×Ascend 910B
  • 目标分支:trainer_dev
  • 并行组合:HSDP+TP、DP+CP+TP、HSDP+TP+EP MoE

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 330
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/330

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing dualmode Trainer parameter handling into FSDP fully_shard, focusing on TPShardMetaInfo, optimizer layout construction, Muon shape restoration, and multidimensional replicate relay. Reproduce the listed HSDP+TP or DP+CP+TP cases, then verify that optimizer updates match the standalone model step by step and that the MoE HSDP+TP+EP path preserves relay dependencies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.