abetlen / abetlen/llama-cpp-python
Serverless inferencing, basic chatbot style
- 主要語言
- Python
- 星號
- 10.6k
- 分支
- 1.4k
- PR 合併指標
- PR 指標待擷取
描述
**Is your feature request related to a problem? Please describe.**
Serverless inferencing is possible with llama.cpp via llama-cli, for example one can do something like this:
llama-cli -m /home/curtine/.local/share/ramalama/models/ollama/granite-code:latest --log-disable --in-prefix --in-suffix --no-display-prompt -p "You are a helpful assistant" -c 2048 -cnv
this can be useful, no idle processes around, now single point of failure, no blocked port, each process can target a separate GPU/CPU, no need for root access to restart some daemon, etc.
**Describe the solution you'd like**
Serverless inferencing support, something similar to llama-cli but integrated with llama-cpp-python.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
貢獻指南
評估
這個 Issue 還沒有評估資料。