CatchTheTornado / CatchTheTornado/text-extract-api
[feat] Ollama concurrent requests
Open
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 279
- PR merge metrics
- No merged PRs in 30d
Description
Ollama by default doesn't support concurrent requests. We need to work on it as it's pretty huge bottleneck for now.
More info: https://www.reddit.com/r/LocalLLaMA/comments/1dt5n6l/ollama_now_runs_inference_concurrently_by_default/?rdt=57766
Maybe we'll need to migrate to vllm - https://github.com/vllm-project/vllm
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.