abetlen / abetlen/llama-cpp-python
When multiple requests are processed, the first request is interrupted
Aberta
bug
- Linguagem predominante
- Python
- Estrelas
- 10.6k
- Forks
- 1.4k
- Métricas de merge de PRs
- Métricas de PR pendentes
Descrição
When multiple requests are processed, the first request is interrupted. How to solve this problem?
My run command is as follows：
python3 -m llama_cpp.server --model ./models/WizardLM-13B-V1.2/ggml-model-f16-Q5.gguf --n_gpu_layers 1 --n_ctx 8192
I tried to set up the following command：
python3 -m llama_cpp.server --model ./models/WizardLM-13B-V1.2/ggml-model-f16-Q5.gguf --n_gpu_layers 1 --n_ctx 8192 --interrupt_requests False
But --interrupt_requests False did not take effect.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.