abetlen / abetlen/llama-cpp-python

[FR] Auto-load/unload models to save memory when unused

Abierto
#736 0 comentarios 20 reacciones 0 asignados Ver en GitHub
enhancement server
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

**Is your feature request related to a problem? Please describe.**
In a shared machine, or even a personal one, it's better to automatically free up the memory used when the model is not in use.

**Describe the solution you'd like**
Some `idle_before_unload=600` parameter that unloads the model after that many seconds of being idle. The model should be reloaded automatically whenever it is used again.

**Additional context**
- https://github.com/abetlen/llama-cpp-python/issues/302

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.