kubeflow / kubeflow/docs-agent

Performance: Synchronous Milvus search blocks the async event loop in execute_tool()

Open
#110 2 comments 0 reactions 1 assignee Claimed by @Kunal-Somani View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

Hi team! I am an aspiring GSoC 2026 contributor auditing the concurrency performance of the chat servers. I noticed a critical bottleneck in the tool execution pipeline that blocks the async event loop.

### Root Cause Analysis
In both `server/app.py` (WebSocket) and `server-https/app.py` (FastAPI), the `execute_tool()` function is defined as `async`, but it directly calls `milvus_search()`, which contains synchronous blocking I/O (PyMilvus `connections.connect`, `collection.load`, and `collection.search`).

Because this synchronous network call is executed directly on the main thread rather than being offloaded, it freezes the Uvicorn/asyncio event loop. Under concurrent load, the server will completely halt all other WebSockets, HTTP connections, and health checks until the Milvus I/O operation finishes.

### Proposed Architecture Fix
We need to offload the synchronous database call to a background worker thread.

**In `server/app.py` (Standard Asyncio):**
Modify the call in `execute_tool` to use `asyncio.to_thread`:
```python
result = await asyncio.to_thread(milvus_search, query, top_k)
```

**In `server-https/app.py` (FastAPI):**
Utilize FastAPI's built-in threadpool concurrency:
```python
from fastapi.concurrency import run_in_threadpool
result = await run_in_threadpool(milvus_search, query, top_k)
```

### Impact
This small change ensures the main event loop remains unblocked, allowing the servers to handle concurrent LLM streaming and multiple users while waiting for the vector database to return results.

If the core team agrees this optimization is necessary for production, I have a local fix prepared and would be happy to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.