mindspore-ai / mindspore-ai/hyper-parallel

MindSpore replicate_params backward prefetch ST teardown 阶段 segfault

Open
#253 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

问题描述

HyperParallel 门禁流水线中,MindSpore fully_shard replicate_params backward prefetch 回归用例在主体执行通过后,msrun worker 在进程退出/GC 阶段触发 native segmentation fault,导致流水线失败。

失败流水线:
https://build.mindspore.cn/blue/organizations/jenkins/Hyper-parallel_Atomgit_Gate/detail/Hyper-parallel_Atomgit_Gate/4372/pipeline

失败用例:
tests/mindspore/st/fully_shard/test_fully_shard_replicate_prefetch_regression.py::test_ms_fully_shard_replicate_prefetch_regression_suite

实际失败的子用例:
test_ms_fully_shard_replicate_params_backward_prefetch_regression

关键日志

_test_fully_shard_replicate_prefetch_regression.py::test_ms_fully_shard_replicate_params_backward_prefetch_regression
======================= 1 passed, 30 warnings in 38.07s ========================
Fatal Python error: Segmentation fault
Current thread ...:
  Garbage-collecting
  <no Python frame>

Worker process 24341 exit with exception. Error code: -11.
RuntimeError: Distributed job exited with exception.
AssertionError: List cases failed: ['test_ms_fully_shard_replicate_params_backward_prefetch_regression']

同一 suite 中 test_ms_hsdp_replicate_dtensor_state_visible_after_backward 已通过,并打印:

rank: 0, max_logits_sum_after_backward: 112.0

test_ms_fully_shard_replicate_params_backward_prefetch_regression 也已打印 loss:

rank: 0, regression step loss: 2.7763614654541016

说明失败不是 Python 断言不满足,而是用例结束后的 MindSpore 分布式运行时清理阶段 native crash。

临时规避

PR #851 临时跳过该 ST,避免当前门禁被 teardown 阶段 segfault 阻塞。待 msrun worker 退出阶段 core dump 修复后,需要恢复该用例。

期望结果

  • test_ms_fully_shard_replicate_params_backward_prefetch_regression 执行通过后,msrun worker 正常退出。
  • 外层 parallel_run 不应因 worker -11 退出而失败。
  • 恢复 test_ms_fully_shard_replicate_prefetch_regression_suite 的门禁覆盖。

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 219
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/219

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/mindspore/st/fully_shard/test_fully_shard_replicate_prefetch_regression.py and reproduce test_ms_fully_shard_replicate_params_backward_prefetch_regression through the HyperParallel_Atomgit_Gate pipeline. Check the msrun worker teardown after the passing loss output and compare it with the HSDP case that exits normally. Done means the worker exits without code -11, parallel_run succeeds, and the full regression suite is restored.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.