dotnet / dotnet/TorchSharp

Add empty_cache for releasing GPU memory

Open
#892 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.9k
Forks
228
PR merge metrics
No merged PRs in 30d

Description

When I run the code sample below, my GPU monitoring seems to show that the memory that was allocated during execution (in a jupyter notebook) is still allocated despite the fact that I've done all I know to do to have intermediate tensors disposed

```
#r "nuget:TorchSharp-cuda-linux"
#r "nuget:TorchSharp"

open TorchSharp

let test () =
use d = torch.NewDisposeScope()
use tt = torch.randn(50000,50000,device=torch.device("cuda:7"))
tt.MoveToOuterDisposeScope()

let test2() =
use d2 = torch.NewDisposeScope()
use ttt = test()
()

let empty_result = test2()
```

I get a similar result when I do the same experiment in python with pytorch

```
import torch

def test():
tt = torch.randn(50000,50000,device=torch.device('cuda:7'))
return ttt

def test2():
ttt = test()
return 0

empty_result = test2()
```

but I can free the memory by calling torch.cuda.empty_cache(). @NiklasGustafsson says

> The underlying library does keep a high-watermark of allocated GPU memory, so even when you dispose of tensors, the overall allocation won't necessary go down. I'll see how I can get empty_cache() implemented

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.