kubeflow / kubeflow/docs-agent

Performance: SentenceTransformer reload + Milvus reconnect compound to ~3s overhead per search in main servers

Open
#183 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

## Summary

Both `server/app.py` and `server-https/app.py` instantiate a new `SentenceTransformer` and open a new Milvus connection inside the `milvus_search` function on every search call. These two costs compound to roughly 2-3 seconds of overhead per query on CPU before the actual vector search even begins.

Issues #63 and #28 each track one half of this individually. This issue consolidates them as a compound performance problem since fixing only one still leaves significant per-request overhead.

## Location

**In `server/app.py` and `server-https/app.py`** -- `milvus_search()` function:

```python
def milvus_search(query, top_k=5):
# Cost 1: New Milvus connection per request (~200-500ms)
connections.connect(alias="default", host=MILVUS_HOST, port=MILVUS_PORT)
collection = Collection(MILVUS_COLLECTION)
collection.load() # idempotent after first call

# Cost 2: Model reload from disk per request (~2-3s on CPU)
encoder = SentenceTransformer(EMBEDDING_MODEL)
```

## The Right Pattern Already Exists

`kagent-feast-mcp/mcp-server/server.py` already implements the correct pattern:

```python
model: SentenceTransformer = None
client: MilvusClient = None

def _init():
global model, client
if model is None:
model = SentenceTransformer(EMBEDDING_MODEL)
if client is None:
client = MilvusClient(uri=MILVUS_URI, ...)
```

The main servers should adopt this same lazy-init singleton pattern. Combined, this would reduce per-query overhead from ~3s to near zero for all requests after the first.

Note: As Sinan pointed out in the Slack discussion, `collection.load()` is idempotent server-side in Milvus -- once loaded it stays loaded across client disconnects. So the real per-request costs are the model reload and the connection setup/teardown, not all three.

PR freeze is on, so flagging this for when PRs reopen. Happy to pick this up.

Related: #63 (model reload), #28 (connection pooling), #181 (content truncation)

Contributor guide

Open the contributing guide

Research direction

Start with milvus_search() in server/app.py and server-https/app.py, then compare their initialization with kagent-feast-mcp/mcp-server/server.py. The work is done when repeated searches reuse the SentenceTransformer and Milvus client or connection in both main servers, leaving initialization overhead only on the first search.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.