mindspore-ai / mindspore-ai/hyper-parallel
build #158 门禁失败:FSDP+EP loss 放大、MRS output_m 顺序错误、worker 路径
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
该问题是怎么引起的?
Jenkins build #158 门禁失败,涉及 3 类问题:
-
FSDP+EP 训练 loss 放大(Torch)— EP AlltoAll 的 token 重复放大 + fully_shard 对 DTensor 参数的 SUM reduce-op 导致 loss 不匹配。
-
MatmulReduceScatter output_m 元组顺序错误(MindSpore)— DP+TP 联合分片时 output_m tensor_map 写成 (comm_mesh_dim, dp_mesh_dim),正确应为 (dp_mesh_dim, comm_mesh_dim),导致 full_tensor() 重建行序异常。
-
msrun worker 文件路径(MindSpore)— test_moe.py 使用相对路径引用 worker,msrun 从项目根目录运行 pytest 时找不到。
重现步骤
在 Ascend 910B 环境下运行 build #158 对应门禁测试。
报错信息
- FSDP+EP: standalone 与 FSDP+EP 的 loss 不匹配
- MRS: full_tensor() 重建错误,max_diff ~120.5
- MoE: msrun 找不到 worker 文件
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 229
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/229
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the build #158 gate tests on Ascend 910B and inspect the three reported failures: FSDP+EP loss mismatch, MatmulReduceScatter full_tensor() reconstruction, and the worker lookup in test_moe.py. Done means the loss matches the standalone result, MRS reconstruction has the correct row order, and msrun can locate the worker file from the project root.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100