aiondemand / aiondemand/aiod-enhanced-interaction
Make the whole backend more asynchronous
- Ngôn ngữ chính
- Python
- Star
- 0
- Fork
- 2
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
The idea behind this task is to make each thread accommodate multiple tasks that would alternate between each other, making the main backend processes more efficient. Currently the processing of user queries as well as calculating embeddings of new AIoD assets are implemented in a sequential manner which is sufficient for now, but is not scalable once the user traffic substantially increases.
### Proposed changes
- **Search thread** that focuses on processing of user queries
- Allow for processing of multiple user queries at the same time (limit the number of tasks with semaphore akin to how tasks associated with experiment runs are processed on RAIL backend)
- Make the main services on backend asynchronous to facilitate concurrency
- Milvus service -> `AsyncMilvusClient`
- HTTP requests (getting assets from AIoD)
- Ollama LLM invocations -> `.ainvoke` function
- HuggingFace model -> either read up on how to make models async, or move the model to a separate thread?
- **Embedding thread** that focuses on updating the Milvus database to reflect changes on AIoD platform
- Currently this thread sequentially alternates between the following four logical steps described below. In our efforts to make our application more concurrent, we should separate these steps into their own tasks, make them run concurrently:
- HTTP requests to retrieve assets
- Extraction of metadata using an Ollama LLM
- Computation of model embedding
- Storing of new embedding in Milvus
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.