mindspore-ai / mindspore-ai/hyper-parallel
[Bug]: fully_shard 在 reduce_dtype 与梯度 dtype 相同时释放了调用方的梯度 storage,导致 ConcatD 访问非法 GM 地址
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
Checklist
- 1. I have searched the existing issues.
- 2. I have read the relevant documentation.
- 3. I have created a minimal reproduction case that clearly demonstrates the issue.
🐛 Describe the bug
fully_shard 在 reduce-scatter 完成后释放通信输入的 storage,但在 reduce_dtype
与梯度 dtype 相同时,该"通信输入"与调用方的真实梯度共享同一块 storage,导致
释放了 fully_shard 并不拥有的内存。
后果:该内存块被分配器回收再分配给后续算子,训练首步即在 Ascend 上崩溃:
fault kernel_name=ConcatD_36655e11f3e4558425f72c5a4171ecc6_high_performance_4000001
aivec error, error code = 0x800000
errorStr: MTE accesses an invalid GM address or the cross-device memory access times out
崩溃在下一次 host 同步处暴露为 RuntimeError: SyncCopy failed for Tensor(shape=[], dtype=Float32)。
缺陷位置
hyper_parallel/platform/mindspore/fully_shard/param.py
def reduce_scatter_grad(self, ...):
...
self._grad = grad.to(self.reduce_comm_dtype(grad)) # (1) 同 dtype 时不拷贝
...
self._grad = self._grad.reshape(-1) # (2) 仍共享 storage
def reduce_scatter_output(self):
if self.reduce_scatter_comm_ctx.reduce_scatter_handle is not None:
self.reduce_scatter_comm_ctx.reduce_scatter_handle.wait()
self._grad.untyped_storage().resize_(0) # (3) 释放调用方的梯度
reduce_comm_dtype 在 reduce_dtype 已设置时直接返回它。当
MixedPrecisionPolicy(reduce_dtype=float32) 且梯度本身为 fp32 时,(1) 是同 dtype
转换——MindSpore 返回新的 Tensor 对象但共享源 storage,(2) 继续共享,于是 (3)
把真实梯度缓冲的 storage 置零。
仅当 reduce_dtype 与梯度 dtype 不同时才会产生独立副本,因此该缺陷是配置相关的。
最小复现(纯 CPU,无需 NPU)
import numpy as np
import mindspore as ms
def check(make_comm_buffer):
grad = ms.Tensor(np.arange(16, dtype=np.float32).reshape(4, 4))
comm_buf = make_comm_buffer(grad)
comm_buf.untyped_storage().resize_(0) # reduce_scatter_output 所做的事
return grad.untyped_storage().size() == 0 # 调用方的梯度是否被释放
print(check(lambda g: g.to(ms.float32))) # True <- 缺陷
print(check(lambda g: g.to(ms.float32).reshape(-1))) # True <- 缺陷
print(check(lambda g: g.to(ms.bfloat16))) # False <- 真拷贝,安全
注意:不要用 untyped_storage().data_ptr() 判断是否共享——reshape 之后该指针会
变化,具有误导性;resize_ 的传播才是可靠判据。
定位过程
在 mindspore/mindformers 门禁上做三点二分,同一份 mindformers 代码,仅更换 hyper_parallel:
| hyper_parallel 构建 | 含 e85a106a |
含 54793eb2 |
门禁 |
|---|---|---|---|
r1.0.0_20260715000008 |
✗ | ✗ | 全部通过 |
master_20260824010007_454d844f |
✓ | ✗ | 全部通过 |
master_20260827010008_095c6ab6 |
✓ | ✓ | UT 失败 |
绿→红区间内仅 54793eb2「refactor(fully_shard): align MindSpore FSDP lifecycle with
Torch」(MR !1245) 一个提交触及 fully_shard。该提交删除了原先带 .contiguous() 的
pack_utils.py 打包路径,改为在 reduce_scatter_grad 内联 chunk/cat,并新增了
上述 storage 释放。
已排除的因素:
- mindspore/hyper_parallel 版本错配:将 mindspore 由
master_20260630010018
升至配套的master_20260827010013后重跑,fault kernel 名称与前次逐字节相同。 - 调用方缺少同步:在优化器读取梯度前补
hsdp_sync_stream()无效——该接口内部
执行的正是这段释放逻辑,只会让释放更早发生,而非避免。
触发配置
- deepseek3,2 卡,
data_parallel_shard: -1、data_parallel_shard_strategy: optim_grads_params MixedPrecisionPolicy(reduce_dtype=float32, apply_grad_on_fp32_main_grad=True)- 模型
params_dtype: float32(梯度为 fp32,与reduce_dtype相同) comm_fusion=False
建议修复方向
在释放前确保 self._grad 是 fully_shard 自有的缓冲,例如在 (1) 处当转换未产生新
storage 时显式 clone(),或在 (3) 处仅释放确认由本模块分配的 storage。
Expected behavior
fully_shard 只应释放自身分配的通信缓冲。当 grad.to(reduce_dtype) 未产生新
storage 时(reduce_dtype 与梯度 dtype 相同),reduce_scatter_output() 不应对该
storage 调用 resize_(0),否则会破坏调用方仍在使用的梯度缓冲。
Environment info
- hyper_parallel: master_20260827010008_095c6ab / master_20260829010008_15a0afec(均复现)
- mindspore: master_20260827010013_b007b0db / master_20260829010012_c5d5d454
- 硬件: Ascend 910B, 2 卡
- 未复现版本: r1.0.0_20260715000008, master_20260824010007_454d844f
- 最小复现脚本为纯 CPU,任意 MindSpore 安装均可运行
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 355
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/355
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in hyper_parallel/platform/mindspore/fully_shard/param.py, focusing on reduce_scatter_grad and reduce_scatter_output, then run the CPU minimal reproduction from the issue. Trace storage sharing when reduce_dtype matches the gradient dtype and compare it with the differing-dtype case. Done means fully_shard releases only its own communication buffer and the reproduction confirms the caller's gradient storage remains usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100