microsoft / microsoft/onnxruntime-genai
.Net How to free GPU memory after each inference
@RyanUnderhill is already working on this.
Since Apr 9, 2025.
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
I am using Phi3.5mini-cuda-fp16 With A Nvida GPU (24G Memory).
When i load model Memory is 8490MiB in use.
When I entered an inference of about 3K tokens, the GPU Memory used 10580MiB
If I continue the conversation afterwards, GPU memory will continue to rise
If I am not having a conversation, even if I leave it for an hour, the memory will not decrease.
I don't know if this is a bug, as this phenomenon seems to have existed since 0.4, and the same goes for 0.5.2
**Or did I miss something?**
This is My code ,I did not forget to release any object, of course, the Model object was not released because we need to reuse it
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.