abetlen / abetlen/llama-cpp-python
Concurrent request handling
Abierto
question
- Lenguaje dominante
- Python
- Estrellas
- 10.6k
- Forks
- 1.4k
- Métricas de merge de PR
- Métricas de PR pendientes
Descripción
Hey there!! 🙏
I am currently working on a project that involves the sending request to the model using flask api and when user sends the request concurrently the model is not able to handle it. Is there any way i can handle multiple concurrent request to the model and serve multiple users at the same time?
Please help! @abetlen
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.