mindspore-ai / mindspore-ai/hyper-parallel
为HP增加训练前预检与训练后诊断标准能力
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
问题背景
在 HyperParallel 实际落地中,以下问题反复出现:
- 训练启动前缺少统一的“环境/依赖/运行时”预检入口。
- 训练后问题定位链路分散,性能、精度、故障日志分析依赖人工经验。
- 不同任务(mindformers/mindone/llamafactory/transformers)排障方式不统一,复用性差。
- Ascend profiler 数据存在,但缺少标准化摘要与瓶颈分类脚本,导致分析成本高、结论不一致。
这会直接导致:
- 问题发现慢,定位周期长;
- 同类问题重复排查;
- 团队间排障口径不一致,难以沉淀可复用流程。
目标
为 HyperParallel 增加一套低侵入、可复用、可脚本化的基础工具能力,参考mindspore-skills:
- 训练前:
preflight预检能力 - 训练后:
diagnose诊断能力(故障/精度/性能三类)
方案范围(本期)
- 新增
scripts/preflight/:单机训练前就绪检查与结构化报告输出。 - 新增
scripts/diagnosis/:- failure:上下文采集与 traceback 摘要
- accuracy:环境/版本采集与指标差异摘要
- performance:Ascend profiler 摘要与瓶颈候选分类
- 新增统一报告契约
report.schema.json。 - 增加
hp-preflight/hp-diagnose技能入口与适配文档,统一使用方式。
为什么现在需要做
- 该能力对现有训练主链路零侵入,但能显著降低排障成本。
- 与当前迭代重点(稳定性、可复现性、性能优化效率)直接相关。
- 先落标准化基础层,可为后续自动化诊断/CI 集成提供统一接口。
验收标准
- 能在常见工作区执行 preflight 并输出结构化结果;
- 能按 symptom 路由到 failure/accuracy/performance 诊断链路;
- 在有 Ascend profiler 产物时可生成性能摘要与瓶颈候选;
- 不影响现有训练功能与测试主路径。
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 113
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/113
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing repository structure and any current skill entry points before planning the new scripts/preflight/ and scripts/diagnosis/ areas. Use report.schema.json and the hp-preflight and hp-diagnose entry points as the integration boundaries. Done means structured preflight output, symptom-based failure/accuracy/performance routing, Ascend profiler summaries when available, and no regressions to existing training tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100