mindspore-ai / mindspore-ai/hyper-parallel

automodels目录重构

Open
#161 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

需求背景

hyper_parallel/auto_models/ 在演进中成为承载模型适配、Trainer、数据、分布式构建与公共组件的单一日录,模块职责混杂、依赖方向不清:models/trainer/data 平级语义无法表达,components 同时装优化器、损失、量化、checkpoint 与数据集,新组件接入位置无规可循;同时存在 legacy core/shard API、旧 hyper_parallel/models 兼容包等多套入口,维护成本高、语义易漂移。

需要一次目录级重构,将 auto_models 拆分为职责单一的顶层包,固化依赖方向,收敛唯一对外接口与唯一训练入口,并保证对外 API、配置契约与训练行为完全不变。

开发设计

顶层拆分:auto_models/ 溶解为 models/(模型族 adapter 层 + api/build_options/replacement/_transformers,对外接口)、components/(modules/functional/quantization/losses/optim/checkpoint)、distributed/、trainer/、data/ 五个顶层包。
固化依赖方向:trainer ──► models ──► components/distributed ──► core/platform,data 仅依赖 torch/transformers;distributed → models 仅允许 build_options 叶子 DTO 及 planner 函数级的 registry 查询。
模型族 adapter 层:Qwen3-MoE 的替换工厂、CP/EP 契约收敛至 models//adapter/,通过 ModelAdapterSpec 声明、目录约定自动发现注册;唯一训练入口收敛为 models/qwen3_moe/recipes/train.yaml,删除 legacy fusion 文件。
构建管线拆分:checkpoint 收尾并入 checkpoint_loader,plan DTO 边界归一化,god module 拆分为 model_builder,registry 拆分为 config_resolver + models/registry。
清理:删除 legacy core/shard API、旧 hyper_parallel/models 与 components 兼容包;references 在 py/yaml/sh/setup.py 及文档中全量改写。
测试先行的表征测试策略(M0):迁移前锁定公共 API 签名/dataclass 字段快照、构建顺序 golden case、训练 YAML 快照与共享 fixtures,迁移中逐阶段保持绿色;门禁环境无法承载的用例(依赖坏掉的 torch_npu/torchvision 传递链、仓库文件系统路径)按类别剔除,保留可独立运行的导入边界探针。
验收标准

对外公共 API 的符号名、inspect.signature、dataclass 字段顺序/默认值及 all 在重构前后一致,有 UT 快照覆盖。
入口包导入无分布式初始化、无网络副作用,导入边界有 UT 守护。
唯一 recipe models/qwen3_moe/recipes/train.yaml 可驱动完整训练,替换/CP/EP 契约 UT 覆盖。
实际集群训练行为不变:910C 16 卡 Qwen3-30B-A3B 训练验证通过。
验证结果

已在独立 editable 安装的公共源码上执行:

HYPER_PARALLEL_PLATFORM=torch PYTHONNOUSERSITE=1 python -m pytest -q tests/ut/
结果:全量 UT 4562 passed;表征测试套件全绿。

已在 910C 集群完成 16 卡 Qwen3-30B-A3B 训练验证,训练正常收敛,行为与重构前一致。

本次 squash 前后代码树一致;分支保持单 commit(58 个阶段提交 squash 为一个)。服务端 CI 与 CodeCheck 待 PR !1328 创建后验证。

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 368
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/368

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the proposed auto_models/ split and the existing hyper_parallel/auto_models/ tree, then run tests/ut/ to establish the current baseline. Trace the public APIs, model-builder and distributed boundaries described in the issue, with models/qwen3_moe/recipes/train.yaml as the training entry point. Done means the stated package layout, API snapshots, import-boundary tests, full unit suite, and training validation remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, distributed-systems, machine-learning
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.