cuda.core: VirtualMemoryResource grow path's Buffer._clear() re-triggers mr.deallocate() on an already-freed VA range

未关闭
#2,877 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
52/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python
领域
backend

调研方向

从 cuda_core/cuda/core/_memory/_virtual_memory_resource.py 中的 _grow_allocation_slow_path 和 _buffer.pyx 中的 Buffer._clear() 开始。运行 test_vmm_allocator_grow_allocation,并检查 Buffer.from_handle() 和 deallocate() 路径,以验证清除旧 buffer 是否会调用其 Deleter。完成的标准是,grow 路径不再尝试 deallocate 已释放的 VA 范围,也不再发出报告中的 CUDAWarning。

由索引模型根据 Issue 内容生成。

描述

bug cuda.core

Summary

_grow_allocation_slow_path (cuda_core/cuda/core/_memory/_virtual_memory_resource.py:362) frees the old buffer's VA range manually (cuMemAddressFree(int(buf.handle), aligned_prev_size), line 457), then calls buf._clear() (line 460) with a comment saying this stops the old buffer's destructor from freeing it again.

Buffer._clear() (_buffer.pyx:262) does self._h_ptr.reset(). For a shared_ptr, reset() isn't a no-op invalidation — if buf holds the last reference, it runs the deleter immediately. This buffer's handle was created with mr=self (Buffer.from_handle(..., mr=self) in allocate()), so its deleter calls back into mr.deallocate() with the old, already-freed pointer and size. deallocate() then calls cuMemRetainAllocationHandle on a VA range that no longer exists, which fails with CUDA_ERROR_INVALID_VALUE. Under the error-handling policy landed in #2759, that failure is caught in the deleter and reported as a CUDAWarning instead of being silent; previously it was silently swallowed.

Evidence

Seen in CI, test_vmm_allocator_grow_allocation:

_virtual_memory_resource.py:292: CUDAWarning: mr.deallocate() failed during Buffer
destruction; the allocation may have leaked: CUDA_ERROR_INVALID_VALUE: This indicates
that one or more of the parameters passed to the API call is not within an acceptable
range of values.

Hypothesis, not yet confirmed by reproduction

Read from the code during review of #2759, not verified by running the test with added instrumentation.

Suggested fix direction

_clear()'s intent here seems to be "drop bookkeeping for a range this code already freed by hand," not "release the handle normally." Those need to be different operations: either give Buffer a way to drop its handle without invoking the resource's deallocate() (e.g. release the underlying pointer from the shared_ptr without running the deleter), or have the grow path swap in a handle whose deleter is already a no-op.

Refs: found during review of #2759.

主要语言
Cython
星标
3.4k
派生
329
平均合并
1 天 21 小时
30 天内合并 PR
113

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NVIDIA/cuda-python 的其他 Issue

查看 NVIDIA/cuda-python 的全部 Issue

相似的 Issue

更多 Backend & API Design Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。