kubeflow / kubeflow/docs-agent
SentenceTransformer model reloaded on every search request
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
## Problem
The `milvus_search()` function in both `server/app.py` (line 77) and `server-https/app.py` (line 123) initializes a new `SentenceTransformer` instance on every call:
```python
encoder = SentenceTransformer(EMBEDDING_MODEL)
```
Loading `sentence-transformers/all-mpnet-base-v2` takes ~2-5 seconds and ~400MB of memory each time. In agentic RAG workflows with multiple tool calls per user turn, this adds 10-25 seconds of unnecessary latency per query.
## Proposed Fix
Initialize the model once at module level and reuse it across all requests.
Contributor guide
Research direction
Read milvus_search() in server/app.py around line 77 and server-https/app.py around line 123, starting with how each entry point initializes the encoder. Verify the search request path, then confirm both entry points reuse one model instance rather than loading it for every request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100