Reducing memory footprint of Llama models for Executorch
@tarun292 is already working on this.
Since Mar 6, 2025.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🚀 The feature, motivation and pitch
Spoke with Ali on Discord and creating this thread based on that discussion.
The model I am currently using is Llama 3.2 3B Instruct SpinQuant. Although it works great for my usecase in terms of functionality, I am concerned about the memory footprint since it consumes 3GB of memory right now.
There are companies out there that help with reducing the memory usage (Pruna AI for example). But they don't support converting the model to Executorch format. Can I get some help from the Executorch team in this regard? The help could be in these forms and I believe all of the Executorch community would benefit from this:
- Support (in terms of scripts etc) for converting a Pruna quantized model to Executorch format that anybody can use.
- Directly make available a Pruna quantized model in Executorch format for everyone to use.
Alternatives
No response
Additional context
No response
RFC (Optional)
No response
cc @mergennachin @cccclai @helunwencser @jackzhxng
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.