mindspore-ai / mindspore-ai/hyper-parallel
[Bug] MindSpore FSDP register_hook 每次前向传播累积导致性能劣化
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
该问题是怎么引起的？
MindSpore FSDP Scheduler 中 _backward_pre_hook 使用 register_hook 注册钩子，
每次前向传播都会在 tensor 上累积新的 hook，未被清理，导致随着训练步数增加性能逐渐劣化。
重现步骤
- 使用 MindSpore 后端启动 FSDP 训练
- 多次执行前向传播
- 观察 hook 数量累积及训练速度下降
报错信息
无明显报错，但训练速度随步数增加显著下降。
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 210
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/210
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the MindSpore FSDP Scheduler and its _backward_pre_hook, then inspect how register_hook is used during forward propagation. Reproduce the issue with repeated FSDP forwards and observe hook counts and training speed. Done means hooks no longer accumulate across forwards and the slowdown is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100