mindspore-ai / mindspore-ai/hyper-parallel

[Bug]: 初始化阶段会有两次权重加载过程

Open
#164 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

Checklist
  • 1. I have searched the existing issues (https://gitcode.com/mindspore/hyper-parallel/issues)
  • 2. I have read the relevant documentation.
  • 3. I have created a minimal reproduction case that clearly demonstrates the issue, including a complete code example and the error message with full traceback and error logs.
🐛 Describe the bug

使用Qwen3模型进行训练,在初始化阶段会有两次权重加载过程:

  1. 第一次是在meta初始化阶段from_pretrained会触发读权重到cpu,大概耗时2~5min,有进度条提醒
  2. 第二次是分片后实际初始化阶段,耗时反而更久10min左右,且没加载进度打印
Expected behavior

首次只做meta初始化获取到model,分片后再加载权重

Additional context
Environment info

Python 3.11.10
torch 2.6.0
transformers 5.13.0

Thanks for contributing 🎉!

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 365
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/365

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Qwen3 training reproduction and trace the initialization path around from_pretrained, comparing the meta-initialization and post-sharding stages. Confirm which stage reads weights and where loading progress is reported. Done means initialization performs only the intended weight load after sharding, without the earlier duplicate load.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.