AlexsJones / AlexsJones/llama-panel
Add support for llama-cpp router mode and llama-swap
- Langage dominant
- JavaScript
- Étoiles
- 69
- Forks
- 7
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
Hi, there again.
Continuation of #1 (new issue).
llama-server implemented a router mode a while ago, when the main process acts as a router starting each model as requested, listing all models configured in a separated [models.ini](https://github.com/ggml-org/llama.cpp/tree/master/tools/server#model-presets) file. This makes switching and maintaining different configurations for each model much simpler, more or less like what llama-swap does.
It would be nice if llama-panel could support both. llama-swap has a web panel, but not a separated desktop app, also as far as I know it does not support llama-server router mode by its own. So it would be nice to have a desktop app that would manage everything in a single sleek app.
Another reason for this is that llama-server also uses a separated models folder where individual GGUFs can be store, apart from HF hub. The hierarchy is explained [here](https://github.com/ggml-org/llama.cpp/tree/master/tools/server#model-sources).
These two environment variables could be used too:
LLAMA_ARG_MODELS_DIR - GGUFs dir
LLAMA_CACHE - usually point to HF hub cache (IIRIC it is how it works by default as of now)
So it would be nice is llama-panel could detect is llama-server is running in router mode when it starts and use it. This can also gracefully start and stop any model individually.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.