AOSSIE-Org / AOSSIE-Org/Devr.AI

FEATURE REQUEST:Fix EmbeddingService blocking the event loop

未关闭
#261 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
102
派生
137
PR 合并指标
30 天内没有已合并 PR

描述

### Is your feature request related to a problem?

- [x] Yes, it is related to a problem

### Describe the feature you'd like

# 🌟Feature Description

Offload blocking embedding work from the async event loop so that `EmbeddingService` does not block other concurrent requests. The service should run CPU/GPU-bound `model.encode()` calls in a thread pool (e.g. via `asyncio.to_thread()`) while keeping the public API async.

## 🔍 Problem Statement

`EmbeddingService` exposes async methods (`get_embedding`, `get_embeddings`, `summarize_user_profile`, `search_similar_profiles`) but internally calls **synchronous** SentenceTransformer code:

- **`self.model.encode(...)`** in `get_embedding` and `get_embeddings` is blocking. It runs on CPU/GPU and does not yield to the event loop.
- While one request is generating embeddings, the entire process is blocked: other HTTP requests, agent tools, and background tasks stall until the encode finishes.
- The LLM call in `summarize_user_profile` correctly uses `await self.llm.ainvoke(...)` and is non-blocking; only the embedding step blocks.

This hurts latency and concurrency wherever the service is used (e.g. `issue_processor.py`, `contributor_recommendation.py`, user profiling).

## 🎯 Expected Outcome

- **Event loop stays responsive** during embedding generation: other async work (API handlers, other embeddings, LLM calls) can run while `model.encode()` runs in a worker thread.
- **No change to the public API**: callers keep using `await embedding_service.get_embedding(...)` and `await embedding_service.get_embeddings(...)`.
- **Implementation approach**: add a synchronous helper that performs `model.encode()` and tensor-to-list conversion; call it from `get_embedding` and `get_embeddings` via `asyncio.to_thread()` (or `loop.run_in_executor()` with a `ThreadPoolExecutor`).
- **Optional**: run model lazy-load in a thread at first use to avoid blocking on first request.

## 📷 Screenshots and Design Ideas

**Before:** One long embedding request blocks the event loop → other requests wait.

**After:** Embedding runs in a thread pool → event loop continues handling other requests; embedding call still awaited by the original request.

No UI changes; this is a backend concurrency fix.

## 📋 Additional Context

- **File to change:** `backend/app/services/embedding_service/service.py`
- **Consumers:** `app/services/github/user/profiling.py`, `app/services/github/issue_processor.py`, `app/agents/devrel/github/tools/contributor_recommendation.py`
- **Suggested steps:**
1. Add `import asyncio`.
2. Add a sync helper method (e.g. `_encode(texts)`) that calls `self.model.encode(...)` and returns list(s) of floats.
3. In `get_embedding`: replace direct `model.encode` with `await asyncio.to_thread(self._encode, [text])`, then return the single embedding list.
4. In `get_embeddings`: replace direct `model.encode` with `await asyncio.to_thread(self._encode, texts)` and return the list of lists.
- **Verification:** While one request is generating embeddings, trigger another (e.g. health check or simple async endpoint); the second should respond without waiting for the first.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。