mindspore-ai / mindspore-ai/hyper-parallel

为HP增加训练前预检与训练后诊断标准能力

Open
#310 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

问题背景

在 HyperParallel 实际落地中,以下问题反复出现:

  1. 训练启动前缺少统一的“环境/依赖/运行时”预检入口。
  2. 训练后问题定位链路分散,性能、精度、故障日志分析依赖人工经验。
  3. 不同任务(mindformers/mindone/llamafactory/transformers)排障方式不统一,复用性差。
  4. Ascend profiler 数据存在,但缺少标准化摘要与瓶颈分类脚本,导致分析成本高、结论不一致。

这会直接导致:

  • 问题发现慢,定位周期长;
  • 同类问题重复排查;
  • 团队间排障口径不一致,难以沉淀可复用流程。

目标

为 HyperParallel 增加一套低侵入、可复用、可脚本化的基础工具能力,参考mindspore-skills:

  • 训练前:preflight 预检能力
  • 训练后:diagnose 诊断能力(故障/精度/性能三类)

方案范围(本期)

  1. 新增 scripts/preflight/:单机训练前就绪检查与结构化报告输出。
  2. 新增 scripts/diagnosis/
    • failure:上下文采集与 traceback 摘要
    • accuracy:环境/版本采集与指标差异摘要
    • performance:Ascend profiler 摘要与瓶颈候选分类
  3. 新增统一报告契约 report.schema.json
  4. 增加 hp-preflight / hp-diagnose 技能入口与适配文档,统一使用方式。

为什么现在需要做

  • 该能力对现有训练主链路零侵入,但能显著降低排障成本。
  • 与当前迭代重点(稳定性、可复现性、性能优化效率)直接相关。
  • 先落标准化基础层,可为后续自动化诊断/CI 集成提供统一接口。

验收标准

  • 能在常见工作区执行 preflight 并输出结构化结果;
  • 能按 symptom 路由到 failure/accuracy/performance 诊断链路;
  • 在有 Ascend profiler 产物时可生成性能摘要与瓶颈候选;
  • 不影响现有训练功能与测试主路径。

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 113
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/113

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing repository structure and any current skill entry points before planning the new scripts/preflight/ and scripts/diagnosis/ areas. Use report.schema.json and the hp-preflight and hp-diagnose entry points as the integration boundaries. Done means structured preflight output, symptom-based failure/accuracy/performance routing, Ascend profiler summaries when available, and no regressions to existing training tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.