mindspore-ai / mindspore-ai/hyper-parallel
automodels目录重构
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
需求背景
hyper_parallel/auto_models/ 在演进中成为承载模型适配、Trainer、数据、分布式构建与公共组件的单一日录,模块职责混杂、依赖方向不清:models/trainer/data 平级语义无法表达,components 同时装优化器、损失、量化、checkpoint 与数据集,新组件接入位置无规可循;同时存在 legacy core/shard API、旧 hyper_parallel/models 兼容包等多套入口,维护成本高、语义易漂移。
需要一次目录级重构,将 auto_models 拆分为职责单一的顶层包,固化依赖方向,收敛唯一对外接口与唯一训练入口,并保证对外 API、配置契约与训练行为完全不变。
开发设计
顶层拆分:auto_models/ 溶解为 models/(模型族 adapter 层 + api/build_options/replacement/_transformers,对外接口)、components/(modules/functional/quantization/losses/optim/checkpoint)、distributed/、trainer/、data/ 五个顶层包。
固化依赖方向:trainer ──► models ──► components/distributed ──► core/platform,data 仅依赖 torch/transformers;distributed → models 仅允许 build_options 叶子 DTO 及 planner 函数级的 registry 查询。
模型族 adapter 层:Qwen3-MoE 的替换工厂、CP/EP 契约收敛至 models//adapter/,通过 ModelAdapterSpec 声明、目录约定自动发现注册;唯一训练入口收敛为 models/qwen3_moe/recipes/train.yaml,删除 legacy fusion 文件。
构建管线拆分:checkpoint 收尾并入 checkpoint_loader,plan DTO 边界归一化,god module 拆分为 model_builder,registry 拆分为 config_resolver + models/registry。
清理:删除 legacy core/shard API、旧 hyper_parallel/models 与 components 兼容包;references 在 py/yaml/sh/setup.py 及文档中全量改写。
测试先行的表征测试策略(M0):迁移前锁定公共 API 签名/dataclass 字段快照、构建顺序 golden case、训练 YAML 快照与共享 fixtures,迁移中逐阶段保持绿色;门禁环境无法承载的用例(依赖坏掉的 torch_npu/torchvision 传递链、仓库文件系统路径)按类别剔除,保留可独立运行的导入边界探针。
验收标准
对外公共 API 的符号名、inspect.signature、dataclass 字段顺序/默认值及 all 在重构前后一致,有 UT 快照覆盖。
入口包导入无分布式初始化、无网络副作用,导入边界有 UT 守护。
唯一 recipe models/qwen3_moe/recipes/train.yaml 可驱动完整训练,替换/CP/EP 契约 UT 覆盖。
实际集群训练行为不变:910C 16 卡 Qwen3-30B-A3B 训练验证通过。
验证结果
已在独立 editable 安装的公共源码上执行:
HYPER_PARALLEL_PLATFORM=torch PYTHONNOUSERSITE=1 python -m pytest -q tests/ut/
结果:全量 UT 4562 passed;表征测试套件全绿。
已在 910C 集群完成 16 卡 Qwen3-30B-A3B 训练验证,训练正常收敛,行为与重构前一致。
本次 squash 前后代码树一致;分支保持单 commit(58 个阶段提交 squash 为一个)。服务端 CI 与 CodeCheck 待 PR !1328 创建后验证。
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 368
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/368
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the proposed auto_models/ split and the existing hyper_parallel/auto_models/ tree, then run tests/ut/ to establish the current baseline. Trace the public APIs, model-builder and distributed boundaries described in the issue, with models/qwen3_moe/recipes/train.yaml as the training entry point. Done means the stated package layout, API snapshots, import-boundary tests, full unit suite, and training validation remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, distributed-systems, machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100