mindspore-ai / mindspore-ai/hyper-parallel
MindSpore replicate_params backward prefetch ST teardown 阶段 segfault
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
问题描述
HyperParallel 门禁流水线中,MindSpore fully_shard replicate_params backward prefetch 回归用例在主体执行通过后,msrun worker 在进程退出/GC 阶段触发 native segmentation fault,导致流水线失败。
失败用例:
tests/mindspore/st/fully_shard/test_fully_shard_replicate_prefetch_regression.py::test_ms_fully_shard_replicate_prefetch_regression_suite
实际失败的子用例:
test_ms_fully_shard_replicate_params_backward_prefetch_regression
关键日志
_test_fully_shard_replicate_prefetch_regression.py::test_ms_fully_shard_replicate_params_backward_prefetch_regression
======================= 1 passed, 30 warnings in 38.07s ========================
Fatal Python error: Segmentation fault
Current thread ...:
Garbage-collecting
<no Python frame>
Worker process 24341 exit with exception. Error code: -11.
RuntimeError: Distributed job exited with exception.
AssertionError: List cases failed: ['test_ms_fully_shard_replicate_params_backward_prefetch_regression']
同一 suite 中 test_ms_hsdp_replicate_dtensor_state_visible_after_backward 已通过,并打印:
rank: 0, max_logits_sum_after_backward: 112.0
test_ms_fully_shard_replicate_params_backward_prefetch_regression 也已打印 loss:
rank: 0, regression step loss: 2.7763614654541016
说明失败不是 Python 断言不满足,而是用例结束后的 MindSpore 分布式运行时清理阶段 native crash。
临时规避
PR #851 临时跳过该 ST,避免当前门禁被 teardown 阶段 segfault 阻塞。待 msrun worker 退出阶段 core dump 修复后,需要恢复该用例。
期望结果
test_ms_fully_shard_replicate_params_backward_prefetch_regression执行通过后,msrun worker 正常退出。- 外层
parallel_run不应因 worker-11退出而失败。 - 恢复
test_ms_fully_shard_replicate_prefetch_regression_suite的门禁覆盖。
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 219
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/219
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/mindspore/st/fully_shard/test_fully_shard_replicate_prefetch_regression.py and reproduce test_ms_fully_shard_replicate_params_backward_prefetch_regression through the HyperParallel_Atomgit_Gate pipeline. Check the msrun worker teardown after the passing loss output and compare it with the HSDP case that exits normally. Done means the worker exits without code -11, parallel_run succeeds, and the full regression suite is restored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100