Model is twice as large on first load
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Hey, I am trying to load an 8b int8 model on my device with 16 GB of RAM. The model should only be taking slightly over 8 GB of memory but it seems that the during the loading process, the model is being copied, which doubles the memory usage to over 16 GB and causing an OOM.
Is it not possible to stream the model instead?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name a source file, test, or entry point. Start by locating the model-loading path and reproduce loading an 8b int8 model on a device with 16 GB of RAM. Done means the loading process no longer requires more than the available memory while preserving successful model loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100