mindspore-ai / mindspore-ai/hyper-parallel

[Bug]: training demo loses indexed data flow after pretrained model integration

Open Beginner friendly
#187 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

Checklist
  • 已检索现有 Issue,并确认 #326 是 Trainer 端到端总 RFC,不覆盖本次配置回退。
  • 已阅读 training demo、数据组件及 PR1154、PR1155、PR1159 的相关改动。
  • 已通过当前 trainer_dev 配置与三次合入结果的对比复现问题。
🐛 Describe the bug

trainer_dev 依次合入 PR1154、PR1155 和 PR1159 后,examples/training_demo/train.yaml 没有保留 PR1154 引入的 indexed data flow:

  1. PR1154 为 training demo 接入 LLM data transform、indexed mock dataset、indexed dataloader 和 get_batch
  2. PR1155 为验证 TP+FSDP optimizer sharding spec,临时将 demo 改为 tiny Llama、TP=2/FSDP=2、DummyDataset 和 identity transform。
  3. PR1159 恢复 Qwen3-30B-A3B 预训练权重加载、8 卡 FSDP、Muon、activation checkpoint 和健壮化 run.sh,但保留了 PR1155 的临时 DummyDataset 数据链路。

因此当前默认 demo 虽然可以覆盖预训练模型加载和 FSDP 训练,却不再覆盖 Trainer 的标准 indexed dataset 构建、数据变换、dataloader 与 get_batch 链路。

当前实际配置为:

  • train_iters=2eval_iters=0
  • IdentityDataTransform
  • DummyDataset
  • legacy components.data.dataloader.DataLoader
  • 未配置 get_batch

本问题与 PR1155 修复的 optimizer layout 问题相互独立。PR1155 的 fully_shard/param.py、resolver、UT 和 accuracy 看护已经合入;PR1157 revert 已关闭且未进入 trainer_dev,本问题不应重复修改这些文件。

Expected behavior

training demo 应组合保留 PR1154 与 PR1159 的能力:

  • 保留 PR1159 的 Qwen3-30B-A3B 预训练权重加载、8 卡 FSDP、Muon、activation checkpoint 和 run.sh
  • 恢复 PR1154 的 build_llm_data_transform
  • 恢复 build_llm_dataset 的 indexed mock data 配置;
  • 恢复 hyper_models.components.datasets.DataLoader
  • 恢复 build_llm_get_batch
  • 恢复 train_iters=10eval_iters=1

修复范围应限制为 examples/training_demo/train.yaml,不修改 fully_shard/param.py、resolver 或 PR1155 的测试文件。

Additional context
  • 关联总 RFC:#326。
  • 提议配置已在 8×Ascend NPU 上完成 Qwen3-30B-A3B、FSDP=8、Muon 的 10/10 steps 验证,命令退出码为 0。
  • 最终 step:loss=0.00582964grad_norm=0.859375;峰值显存 40.18 GB。
  • 全量 UT 结果为 4123 passed, 207 skipped, 21 failed, 106 subtests passed;21 项失败均可在相同 upstream/trainer_dev 干净基线复现。
Environment info
  • 目标分支:trainer_dev@d253907a
  • 平台:PyTorch,HYPER_PARALLEL_PLATFORM=torch
  • 硬件:8×Ascend 910B
  • 模型:Qwen3-30B-A3B
  • 并行方式:8-way FSDP,TP=CP=EP=PP=1
  • 优化器:Muon

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 332
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/332

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/training_demo/train.yaml and compare its current data components with the indexed flow introduced by PR1154, while preserving the model and distributed settings from PR1159. Confirm the referenced builders and iteration counts, then validate the demo using the documented 8× Ascend configuration; done means the indexed dataset, transform, dataloader, get_batch, and 10/1 training loop are restored without changing optimizer files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.