AnswerDotAI / AnswerDotAI/byaldi

Model is not being offloaded from VRAM

Open
#35 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
851
Forks
91
PR merge metrics
No merged PRs in 30d

Description

I am trying to run the model in Jupyter notebook.

![image](https://github.com/user-attachments/assets/6650c6e1-b4d7-47f9-854c-c93cb7f041e1)
1. In the above iteration I haven't initialized the model.

![image](https://github.com/user-attachments/assets/79a37b66-e78d-4279-b606-727b77febe0f)
2. Now I run the cell the model is loaded and it is showing 6GB of vram occupied right.

![image](https://github.com/user-attachments/assets/79e9e8fe-d40b-46be-8b67-4d9b3b89bfee)
3. Now when I run the cell again the vram usage is doubled.
4. In the consequent runs the model is not occupying more than 12GB but what's interesting thing I have observed is when I am running that inside a loop for suppose I want to create an Index for each file I have, I don't have any other option than do this but this is causing the model to give me vram issues. How do I remove them from vram, I tried torch cuda cache free, tried to delete the variable none isn't working for me. Can you please help or is there something I am doing wrongly ?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.