kubeflow / kubeflow/docs-agent
perf: SentenceTransformer model reloaded on every search request
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
Bug Description
In both server/app.py and server-https/app.py, the SentenceTransformer
model is instantiated inside milvus_search(), meaning it is loaded from disk
on every single query:
def milvus_search(query: str, top_k: int = 5):
encoder = SentenceTransformer(EMBEDDING_MODEL) # ← re-loaded every call
query_vec = encoder.encode(query).tolist()
Impact
Loading a transformer model takes 500ms–2s per call depending on hardware.
In an agentic RAG workflow where search_kubeflow_docs may be invoked
multiple times per conversation turn, this compounds into significant latency.
Proposed Fix
Move the encoder to a module-level singleton initialized once at startup:
# Module level — loaded once
encoder = SentenceTransformer(EMBEDDING_MODEL)
def milvus_search(query: str, top_k: int = 5):
query_vec = encoder.encode(query).tolist() # reuses loaded model
...
Files Affected
server/app.pyserver-https/app.py
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in server/app.py and server-https/app.py, reading milvus_search() and the current SentenceTransformer initialization. Make the encoder load once rather than per request, then verify that repeated searches in both server paths reuse it and still return query results without reloading the model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100