mindspore-ai / mindspore-ai/hyper-parallel
[Bug]: training demo loses indexed data flow after pretrained model integration
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
Checklist
- 已检索现有 Issue,并确认 #326 是 Trainer 端到端总 RFC,不覆盖本次配置回退。
- 已阅读 training demo、数据组件及 PR1154、PR1155、PR1159 的相关改动。
- 已通过当前
trainer_dev配置与三次合入结果的对比复现问题。
🐛 Describe the bug
trainer_dev 依次合入 PR1154、PR1155 和 PR1159 后,examples/training_demo/train.yaml 没有保留 PR1154 引入的 indexed data flow:
- PR1154 为 training demo 接入 LLM data transform、indexed mock dataset、indexed dataloader 和
get_batch。 - PR1155 为验证 TP+FSDP optimizer sharding spec,临时将 demo 改为 tiny Llama、TP=2/FSDP=2、
DummyDataset和 identity transform。 - PR1159 恢复 Qwen3-30B-A3B 预训练权重加载、8 卡 FSDP、Muon、activation checkpoint 和健壮化
run.sh,但保留了 PR1155 的临时DummyDataset数据链路。
因此当前默认 demo 虽然可以覆盖预训练模型加载和 FSDP 训练,却不再覆盖 Trainer 的标准 indexed dataset 构建、数据变换、dataloader 与 get_batch 链路。
当前实际配置为:
train_iters=2、eval_iters=0;IdentityDataTransform;DummyDataset;- legacy
components.data.dataloader.DataLoader; - 未配置
get_batch。
本问题与 PR1155 修复的 optimizer layout 问题相互独立。PR1155 的 fully_shard/param.py、resolver、UT 和 accuracy 看护已经合入;PR1157 revert 已关闭且未进入 trainer_dev,本问题不应重复修改这些文件。
Expected behavior
training demo 应组合保留 PR1154 与 PR1159 的能力:
- 保留 PR1159 的 Qwen3-30B-A3B 预训练权重加载、8 卡 FSDP、Muon、activation checkpoint 和
run.sh; - 恢复 PR1154 的
build_llm_data_transform; - 恢复
build_llm_dataset的 indexed mock data 配置; - 恢复
hyper_models.components.datasets.DataLoader; - 恢复
build_llm_get_batch; - 恢复
train_iters=10、eval_iters=1。
修复范围应限制为 examples/training_demo/train.yaml,不修改 fully_shard/param.py、resolver 或 PR1155 的测试文件。
Additional context
- 关联总 RFC:#326。
- 提议配置已在 8×Ascend NPU 上完成 Qwen3-30B-A3B、FSDP=8、Muon 的 10/10 steps 验证,命令退出码为 0。
- 最终 step:
loss=0.00582964,grad_norm=0.859375;峰值显存 40.18 GB。 - 全量 UT 结果为
4123 passed, 207 skipped, 21 failed, 106 subtests passed;21 项失败均可在相同upstream/trainer_dev干净基线复现。
Environment info
- 目标分支:
trainer_dev@d253907a - 平台:PyTorch,
HYPER_PARALLEL_PLATFORM=torch - 硬件:8×Ascend 910B
- 模型:Qwen3-30B-A3B
- 并行方式:8-way FSDP,TP=CP=EP=PP=1
- 优化器:Muon
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 332
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/332
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/training_demo/train.yaml and compare its current data components with the indexed flow introduced by PR1154, while preserving the model and distributed settings from PR1159. Confirm the referenced builders and iteration counts, then validate the demo using the documented 8× Ascend configuration; done means the indexed dataset, transform, dataloader, get_batch, and 10/1 training loop are restored without changing optimizer files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100