mindspore-ai / mindspore-ai/hyper-parallel
[Bug]: 初始化阶段会有两次权重加载过程
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
Checklist
- 1. I have searched the existing issues (https://gitcode.com/mindspore/hyper-parallel/issues)
- 2. I have read the relevant documentation.
- 3. I have created a minimal reproduction case that clearly demonstrates the issue, including a complete code example and the error message with full traceback and error logs.
🐛 Describe the bug
使用Qwen3模型进行训练,在初始化阶段会有两次权重加载过程:
- 第一次是在meta初始化阶段from_pretrained会触发读权重到cpu,大概耗时2~5min,有进度条提醒
- 第二次是分片后实际初始化阶段,耗时反而更久10min左右,且没加载进度打印
Expected behavior
首次只做meta初始化获取到model,分片后再加载权重
Additional context
Environment info
Python 3.11.10
torch 2.6.0
transformers 5.13.0
Thanks for contributing 🎉!
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 365
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/365
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Qwen3 training reproduction and trace the initialization path around from_pretrained, comparing the meta-initialization and post-sharding stages. Confirm which stage reads weights and where loading progress is reported. Done means initialization performs only the intended weight load after sharding, without the earlier duplicate load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100