kubeflow / kubeflow/docs-agent
Performance: Synchronous Milvus search blocks the async event loop in execute_tool()
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
Hi team! I am an aspiring GSoC 2026 contributor auditing the concurrency performance of the chat servers. I noticed a critical bottleneck in the tool execution pipeline that blocks the async event loop.
### Root Cause Analysis
In both `server/app.py` (WebSocket) and `server-https/app.py` (FastAPI), the `execute_tool()` function is defined as `async`, but it directly calls `milvus_search()`, which contains synchronous blocking I/O (PyMilvus `connections.connect`, `collection.load`, and `collection.search`).
Because this synchronous network call is executed directly on the main thread rather than being offloaded, it freezes the Uvicorn/asyncio event loop. Under concurrent load, the server will completely halt all other WebSockets, HTTP connections, and health checks until the Milvus I/O operation finishes.
### Proposed Architecture Fix
We need to offload the synchronous database call to a background worker thread.
**In `server/app.py` (Standard Asyncio):**
Modify the call in `execute_tool` to use `asyncio.to_thread`:
```python
result = await asyncio.to_thread(milvus_search, query, top_k)
```
**In `server-https/app.py` (FastAPI):**
Utilize FastAPI's built-in threadpool concurrency:
```python
from fastapi.concurrency import run_in_threadpool
result = await run_in_threadpool(milvus_search, query, top_k)
```
### Impact
This small change ensures the main event loop remains unblocked, allowing the servers to handle concurrent LLM streaming and multiple users while waiting for the vector database to return results.
If the core team agrees this optimization is necessary for production, I have a local fix prepared and would be happy to submit a PR!
Contributor guide
Assessment
This issue has not been assessed yet.