inducer / inducer/pycuda

Finer control over GPU memory (access to cu*Destroy from Python)

Open
#370 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2.1k
Forks
298
Avg merge
4m
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe.**
We recently identified a (GPU) memory leak in a routine that creates a new cuda stream on a given context every time it is called. Even though the scope of the `stream` variable is constrained to this function, it doesn't seem to be garbage collected. Every N times the function is called, the total GPU memory allocated (measured through `mem_get_info()` or `nvidia-smi`) increases by a small amount (e.g. 2MB with N=16). After enough times, we completely starve our GPU's memory. See details on a minimal example to replicate in the **Additional context** section.

**Describe the solution you'd like**
I'd like the destructor of `Stream` objects to be exposed (e.g. through an explicit `destroy` method), so that we could force free that memory, or a mechanism to ensure streams out of scope are automatically freed.

**Describe alternatives you've considered**
So far, the only approach that seems to get the job done (although not ideal) is to detach the context and create a new one every once in a while. However, this has latency and performance drops associated with it, and involves having to reinitialize any other tasks that were running in that context.

**Additional context**
Minimal example to reproduce:
```python
import gc
from pycuda import driver as cuda
cuda.init()
ctx = cuda.Device(0).make_context()

def create_streams(N=16):
return [cuda.Stream() for _ in range(N)]

def print_memory():
#ctx.push()
free, total = cuda.mem_get_info()
print(f"Used: {(total - free) / (1024**2)}MB")
#ctx.pop()

print_memory() # x MB
# Allocate some streams
streams = create_streams(N=10)
print_memory() # x+2 MB
# Allocate some more
streams.extend(create_streams(N=20))
print_memory() # x+4 MB
# Delete all streams
del streams
gc.collect() # Even trying to force the garbage collector won't do anything
print_memory() # Still x+4 MB instead of x MB :(

ctx.pop()
```
prints out:
```
Used: 5651.25MB
Used: 5653.25MB
Used: 5655.25MB
Used: 5655.25MB
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.