abetlen / abetlen/llama-cpp-python
[FR] Auto-load/unload models to save memory when unused
Offen
enhancement
server
- Vorherrschende Sprache
- Python
- Sterne
- 10.6k
- Forks
- 1.4k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
**Is your feature request related to a problem? Please describe.**
In a shared machine, or even a personal one, it's better to automatically free up the memory used when the model is not in use.
**Describe the solution you'd like**
Some `idle_before_unload=600` parameter that unloads the model after that many seconds of being idle. The model should be reloaded automatically whenever it is used again.
**Additional context**
- https://github.com/abetlen/llama-cpp-python/issues/302
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.