deepseek-ai / deepseek-ai/DeepEP

[Question] Is there undefined behavior between the calls to `cudaFree` and `cudaIpcCloseMemHandle`?

Open
#497 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

In `Buffer::destroy`, each rank first closes the 7 external memory handles, then `cudaFree`s the local buffer.

However, the docs for `cudaIpcOpenMemHandle` says it is undefined behavior to `cudaFree` the *exported memory region* before the call to `cudaIpcCloseMemHandle`.

So in an extreme situation (actually I think this may be common) it is possible for rank 2 to close the handle exported from rank 1 long after rank 1 has freed that buffer. This should be technically considered undefined behavior, although I haven't been able to construct a test that exhibits clearly erroneous behavior, e. g. crashes.

Basically I am wondering about the robustness of this code: is it UB? Does it just happen to work? Or it is guaranteed to work given the current CUDA implementations?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.