mindspore-ai / mindspore-ai/hyper-parallel
[fully_shard] 延迟初始化场景下，fully_shard内部报错AllGather不支持metaTensor，且有reset_parameter中不应该setattr parameter到对应module中
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
该问题是怎么引起的？
在调试Meta初始化的场景下，发现预取配置下，fully_shard内部报错AllGather不支持metaTensor，根因在于root module 要将所有的HSDPState进行lazy_init,
此外，HSDPParam.reset_parameters方法中，当输入是普通Tensor时的处理分支，不应该进行settattr parameter 到对应module中，否则会导致优化器引用的参数和sharded_param不是同一个对象，引发严重的精度问题。
重现步骤
报错信息
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 67
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/67
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing fully_shard's lazy initialization for the root module and the HSDPParam.reset_parameters path described in the issue. Verify the prefetch configuration no longer sends meta tensors to AllGather and that ordinary Tensor handling preserves the optimizer's parameter identity instead of replacing it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100