abetlen / abetlen/llama-cpp-python
Concurrent request handling
Aperta
question
- Lingua principale
- Python
- Stelle
- 10.6k
- Fork
- 1.4k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
Hey there!! 🙏
I am currently working on a project that involves the sending request to the model using flask api and when user sends the request concurrently the model is not able to handle it. Is there any way i can handle multiple concurrent request to the model and serve multiple users at the same time?
Please help! @abetlen
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.