aiondemand / aiondemand/aiod-enhanced-interaction

Make the whole backend more asynchronous

Đang mở
#17 0 bình luận 0 reaction 1 người được giao Được @marcel-vesely-kinit nhận Xem trên GitHub
Ngôn ngữ chính
Python
Star
0
Fork
2
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

The idea behind this task is to make each thread accommodate multiple tasks that would alternate between each other, making the main backend processes more efficient. Currently the processing of user queries as well as calculating embeddings of new AIoD assets are implemented in a sequential manner which is sufficient for now, but is not scalable once the user traffic substantially increases.

### Proposed changes
- **Search thread** that focuses on processing of user queries
- Allow for processing of multiple user queries at the same time (limit the number of tasks with semaphore akin to how tasks associated with experiment runs are processed on RAIL backend)
- Make the main services on backend asynchronous to facilitate concurrency
- Milvus service -> `AsyncMilvusClient`
- HTTP requests (getting assets from AIoD)
- Ollama LLM invocations -> `.ainvoke` function
- HuggingFace model -> either read up on how to make models async, or move the model to a separate thread?
- **Embedding thread** that focuses on updating the Milvus database to reflect changes on AIoD platform
- Currently this thread sequentially alternates between the following four logical steps described below. In our efforts to make our application more concurrent, we should separate these steps into their own tasks, make them run concurrently:
- HTTP requests to retrieve assets
- Extraction of metadata using an Ollama LLM
- Computation of model embedding
- Storing of new embedding in Milvus

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.