abetlen / abetlen/llama-cpp-python
Concurrent request handling
Ouverte
question
- Langage dominant
- Python
- Étoiles
- 10.6k
- Forks
- 1.4k
- Métriques de merge des PR
- Métriques de PR en attente
Description
Hey there!! 🙏
I am currently working on a project that involves the sending request to the model using flask api and when user sends the request concurrently the model is not able to handle it. Is there any way i can handle multiple concurrent request to the model and serve multiple users at the same time?
Please help! @abetlen
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.