kubeflow / kubeflow/docs-agent

bug(server-https): SentenceTransformer model reloaded on every query causing severe latency

Open
#157 1 comment 0 reactions 1 assignee Claimed by @Kunal-Somani View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

## Bug Description

`milvus_search()` in `server-https/app.py` initializes a fresh `SentenceTransformer` instance on every single query:
```python
def milvus_search(query: str, top_k: int = 5):
encoder = SentenceTransformer(EMBEDDING_MODEL) # reloaded every call
query_vec = encoder.encode(query).tolist()
```

## Impact

- Every user query reloads ~400MB of model weights from disk
- Adds 2-4 seconds of unnecessary latency per query
- Under concurrent load, multiple threads each reload the full model simultaneously, multiplying memory pressure and risk of OOM crash
- The model is stateless between calls — there is no reason to reload it

## Proposed Fix

Move model initialization to module level so it runs exactly once at server startup and is reused across all requests:
```python
# Module level — runs once at startup
encoder = SentenceTransformer(EMBEDDING_MODEL)

def milvus_search(query: str, top_k: int = 5):
query_vec = encoder.encode(query).tolist() # reuses warm model
```

This is the same fix already applied to `server/app.py` at the module level, making both servers consistent.

I will submit a PR with this fix.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.