Partial weight loading for reduced RAM utilization
Open
@iseeyuan is already working on this.
Since Sep 25, 2024.
module: runtime
triaged
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🚀 The feature, motivation and pitch
I'm trying to use Executorch for running LLM's on low powered(low RAM) devices. I would like all the weights to not be loaded into RAM when the model is loaded initially. As and when the forward pass is happening new weights to be loaded and the model to be executed to improve RAM utilization.
This can be further extended by predicting which sparse weights to load as described in Apple's paper LLM in a flash, which has seen a inference speed-up along with less RAM usage
Link here
@iseeyuan
Alternatives
No response
Additional context
No response
RFC (Optional)
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.