mindspore-ai / mindspore-ai/hyper-parallel

build #158 门禁失败:FSDP+EP loss 放大、MRS output_m 顺序错误、worker 路径

Open
#684 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

该问题是怎么引起的?

Jenkins build #158 门禁失败,涉及 3 类问题:

  1. FSDP+EP 训练 loss 放大(Torch)— EP AlltoAll 的 token 重复放大 + fully_shard 对 DTensor 参数的 SUM reduce-op 导致 loss 不匹配。

  2. MatmulReduceScatter output_m 元组顺序错误(MindSpore)— DP+TP 联合分片时 output_m tensor_map 写成 (comm_mesh_dim, dp_mesh_dim),正确应为 (dp_mesh_dim, comm_mesh_dim),导致 full_tensor() 重建行序异常。

  3. msrun worker 文件路径(MindSpore)— test_moe.py 使用相对路径引用 worker,msrun 从项目根目录运行 pytest 时找不到。

重现步骤

在 Ascend 910B 环境下运行 build #158 对应门禁测试。

报错信息
  • FSDP+EP: standalone 与 FSDP+EP 的 loss 不匹配
  • MRS: full_tensor() 重建错误,max_diff ~120.5
  • MoE: msrun 找不到 worker 文件

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 229
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/229

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the build #158 gate tests on Ascend 910B and inspect the three reported failures: FSDP+EP loss mismatch, MatmulReduceScatter full_tensor() reconstruction, and the worker lookup in test_moe.py. Done means the loss matches the standalone result, MRS reconstruction has the correct row order, and msrun can locate the worker file from the project root.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.