kubeflow / kubeflow/docs-agent
bug(server-https): SentenceTransformer model reloaded on every query causing severe latency
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
## Bug Description
`milvus_search()` in `server-https/app.py` initializes a fresh `SentenceTransformer` instance on every single query:
```python
def milvus_search(query: str, top_k: int = 5):
encoder = SentenceTransformer(EMBEDDING_MODEL) # reloaded every call
query_vec = encoder.encode(query).tolist()
```
## Impact
- Every user query reloads ~400MB of model weights from disk
- Adds 2-4 seconds of unnecessary latency per query
- Under concurrent load, multiple threads each reload the full model simultaneously, multiplying memory pressure and risk of OOM crash
- The model is stateless between calls — there is no reason to reload it
## Proposed Fix
Move model initialization to module level so it runs exactly once at server startup and is reused across all requests:
```python
# Module level — runs once at startup
encoder = SentenceTransformer(EMBEDDING_MODEL)
def milvus_search(query: str, top_k: int = 5):
query_vec = encoder.encode(query).tolist() # reuses warm model
```
This is the same fix already applied to `server/app.py` at the module level, making both servers consistent.
I will submit a PR with this fix.
Contributor guide
Assessment
This issue has not been assessed yet.