kubeflow / kubeflow/docs-agent

perf: SentenceTransformer model reloaded on every search request

Open Beginner friendly
#178 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

Bug Description

In both server/app.py and server-https/app.py, the SentenceTransformer
model is instantiated inside milvus_search(), meaning it is loaded from disk
on every single query:

def milvus_search(query: str, top_k: int = 5):
    encoder = SentenceTransformer(EMBEDDING_MODEL)  # ← re-loaded every call
    query_vec = encoder.encode(query).tolist()

Impact

Loading a transformer model takes 500ms–2s per call depending on hardware.
In an agentic RAG workflow where search_kubeflow_docs may be invoked
multiple times per conversation turn, this compounds into significant latency.

Proposed Fix

Move the encoder to a module-level singleton initialized once at startup:

# Module level — loaded once
encoder = SentenceTransformer(EMBEDDING_MODEL)

def milvus_search(query: str, top_k: int = 5):
    query_vec = encoder.encode(query).tolist()  # reuses loaded model
    ...

Files Affected

  • server/app.py
  • server-https/app.py

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in server/app.py and server-https/app.py, reading milvus_search() and the current SentenceTransformer initialization. Make the encoder load once rather than per request, then verify that repeated searches in both server paths reuse it and still return query results without reloading the model.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.