mindspore-ai / mindspore-ai/hyper-parallel

【bug】MindSpore fully_shard 参数 view 被非重入重计算捕获导致 unpack 报错

Open
#284 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

该问题是怎么引起的?

在 MindFormers pynative 训练中,同时满足以下条件时会触发问题:

  • transformer layer 配置非重入重计算(use_reentrant=False)。
  • transformer layer 内部的 norm 子模块配置 fully_shard
  • backward 阶段由 recompute 重跑 transformer layer forward。

重算 forward 会再次执行 norm 的 fully_shard forward pre hook,并进入 MindSporeHSDPParamV2.reset_sharded_param()。其中用于刷新 sharded local view 的 local_tensor.narrow(...) 会进入自动微分,被非重入 recompute 的 saved tensor hooks 记录为 placeholder。

该 tensor 属于 fully_shard 内部参数 view 维护,并非模型 forward activation。它进入 placeholder 序列后,原始 forward 与重算 forward 的 saved tensor 序列可能不一致,导致 backward unpack 时无法找到匹配的重算 tensor。

重现步骤
  1. 在 MindFormers pynative 配置中,为 transformer layer 启用 full recompute,并对 layer 内部 norm 启用 fully_shard
  2. 使用双卡运行训练:
ASCEND_RT_VISIBLE_DEVICES=1,2 \
PYTHONPATH=/home/fdw/code/hyper-parallel:$PYTHONPATH \
/home/fdw/miniforge3/bin/conda run -n py310 \
bash scripts/msrun_launcher.sh "run_mindformer.py --config pynative.yaml --mode 1" 2
  1. 训练进入首个 step 的 backward 后,非重入重计算恢复 activation 时触发异常。

定位结果:pack 的调用链为:

_forward_pre_hook
-> _hsdp_forward_pre_hook
-> hsdp_state.unshard()
-> hsdp_param.unshard()
-> _get_unsharded_param_data()
-> reset_sharded_param()
-> local_tensor.narrow(...)
报错信息
RuntimeError: Unpack is being triggered for a tensor, make sure to do this only once!

报错位置:

mindspore/common/recompute.py::_CreatePlaceholderHook.unpack

期望行为:fully_shard 内部用于维护 sharded parameter view 的操作不应污染 recompute 的 activation placeholder 序列,同时应保持 local tensor 的 requires_grad 状态不变。

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 164
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/164

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mindspore/common/recompute.py::_CreatePlaceholderHook.unpack and trace the reported call chain through MindSporeHSDPParamV2.reset_sharded_param() and local_tensor.narrow(...). Reproduce with the two-card msrun_launcher.sh command and the non-reentrant recompute plus norm fully_shard configuration. Done means the internal parameter-view operation no longer pollutes the activation placeholder sequence, unpack succeeds, and local_tensor retains its requires_grad state.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.