mindspore-ai / mindspore-ai/hyper-parallel

[Bug]: fully_shard 在 reduce_dtype 与梯度 dtype 相同时释放了调用方的梯度 storage,导致 ConcatD 访问非法 GM 地址

Open
#629 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

Checklist
  • 1. I have searched the existing issues.
  • 2. I have read the relevant documentation.
  • 3. I have created a minimal reproduction case that clearly demonstrates the issue.
🐛 Describe the bug

fully_shard 在 reduce-scatter 完成后释放通信输入的 storage,但在 reduce_dtype
与梯度 dtype 相同时,该"通信输入"与调用方的真实梯度共享同一块 storage,导致
释放了 fully_shard 并不拥有的内存

后果:该内存块被分配器回收再分配给后续算子,训练首步即在 Ascend 上崩溃:

fault kernel_name=ConcatD_36655e11f3e4558425f72c5a4171ecc6_high_performance_4000001
aivec error, error code = 0x800000
errorStr: MTE accesses an invalid GM address or the cross-device memory access times out

崩溃在下一次 host 同步处暴露为 RuntimeError: SyncCopy failed for Tensor(shape=[], dtype=Float32)

缺陷位置

hyper_parallel/platform/mindspore/fully_shard/param.py

def reduce_scatter_grad(self, ...):
    ...
    self._grad = grad.to(self.reduce_comm_dtype(grad))   # (1) 同 dtype 时不拷贝
    ...
    self._grad = self._grad.reshape(-1)                  # (2) 仍共享 storage

def reduce_scatter_output(self):
    if self.reduce_scatter_comm_ctx.reduce_scatter_handle is not None:
        self.reduce_scatter_comm_ctx.reduce_scatter_handle.wait()
        self._grad.untyped_storage().resize_(0)          # (3) 释放调用方的梯度

reduce_comm_dtypereduce_dtype 已设置时直接返回它。当
MixedPrecisionPolicy(reduce_dtype=float32) 且梯度本身为 fp32 时,(1) 是同 dtype
转换——MindSpore 返回新的 Tensor 对象但共享源 storage,(2) 继续共享,于是 (3)
把真实梯度缓冲的 storage 置零。

仅当 reduce_dtype 与梯度 dtype 不同时才会产生独立副本,因此该缺陷是配置相关的。

最小复现(纯 CPU,无需 NPU)
import numpy as np
import mindspore as ms

def check(make_comm_buffer):
    grad = ms.Tensor(np.arange(16, dtype=np.float32).reshape(4, 4))
    comm_buf = make_comm_buffer(grad)
    comm_buf.untyped_storage().resize_(0)          # reduce_scatter_output 所做的事
    return grad.untyped_storage().size() == 0      # 调用方的梯度是否被释放

print(check(lambda g: g.to(ms.float32)))               # True  <- 缺陷
print(check(lambda g: g.to(ms.float32).reshape(-1)))   # True  <- 缺陷
print(check(lambda g: g.to(ms.bfloat16)))              # False <- 真拷贝,安全

注意:不要用 untyped_storage().data_ptr() 判断是否共享——reshape 之后该指针会
变化,具有误导性;resize_ 的传播才是可靠判据。

定位过程

在 mindspore/mindformers 门禁上做三点二分,同一份 mindformers 代码,仅更换 hyper_parallel

hyper_parallel 构建 e85a106a 54793eb2 门禁
r1.0.0_20260715000008 全部通过
master_20260824010007_454d844f 全部通过
master_20260827010008_095c6ab6 UT 失败

绿→红区间内仅 54793eb2「refactor(fully_shard): align MindSpore FSDP lifecycle with
Torch」(MR !1245) 一个提交触及 fully_shard。该提交删除了原先带 .contiguous()
pack_utils.py 打包路径,改为在 reduce_scatter_grad 内联 chunk/cat,并新增了
上述 storage 释放。

已排除的因素:

  • mindspore/hyper_parallel 版本错配:将 mindspore 由 master_20260630010018
    升至配套的 master_20260827010013 后重跑,fault kernel 名称与前次逐字节相同。
  • 调用方缺少同步:在优化器读取梯度前补 hsdp_sync_stream() 无效——该接口内部
    执行的正是这段释放逻辑,只会让释放更早发生,而非避免。
触发配置
  • deepseek3,2 卡,data_parallel_shard: -1data_parallel_shard_strategy: optim_grads_params
  • MixedPrecisionPolicy(reduce_dtype=float32, apply_grad_on_fp32_main_grad=True)
  • 模型 params_dtype: float32(梯度为 fp32,与 reduce_dtype 相同)
  • comm_fusion=False
建议修复方向

在释放前确保 self._grad 是 fully_shard 自有的缓冲,例如在 (1) 处当转换未产生新
storage 时显式 clone(),或在 (3) 处仅释放确认由本模块分配的 storage。

Expected behavior

fully_shard 只应释放自身分配的通信缓冲。当 grad.to(reduce_dtype) 未产生新
storage 时(reduce_dtype 与梯度 dtype 相同),reduce_scatter_output() 不应对该
storage 调用 resize_(0),否则会破坏调用方仍在使用的梯度缓冲。

Environment info
  • hyper_parallel: master_20260827010008_095c6ab / master_20260829010008_15a0afec(均复现)
  • mindspore: master_20260827010013_b007b0db / master_20260829010012_c5d5d454
  • 硬件: Ascend 910B, 2 卡
  • 未复现版本: r1.0.0_20260715000008, master_20260824010007_454d844f
  • 最小复现脚本为纯 CPU,任意 MindSpore 安装均可运行

schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 355
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/355

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with hyper_parallel/platform/mindspore/fully_shard/param.py, especially reduce_scatter_grad and reduce_scatter_output, then run the minimal CPU reproduction to confirm storage sharing when dtypes match. Trace how the communication buffer is created and released; done means fully_shard releases only storage it owns while preserving the caller's gradient storage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.