kubeflow / kubeflow/docs-agent

Performance Bug: SentenceTransformer is re-initialized on every Milvus search request

Open Beginner friendly
#128 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

### Bug Description
In `server/app.py`, the `SentenceTransformer(EMBEDDING_MODEL)` is instantiated inside the `milvus_search()` function. This causes the application to load the entire embedding model into memory on every single tool call, resulting in severe latency spikes (multiple seconds per search) and risking OOM crashes under concurrent WebSocket connections.

### Proposed Solution
Move the `SentenceTransformer` initialization to the global scope so it acts as a singleton loaded once at server startup. The `milvus_search` function should reference this global instance.

I have tested this locally and it drastically reduces tool execution latency. I will open a PR with the fix shortly.

Contributor guide

Open the contributing guide

Research direction

Inspect server/app.py, starting with milvus_search() and the SentenceTransformer(EMBEDDING_MODEL) construction. Move initialization to global scope, have milvus_search() use that instance, and verify searches no longer re-initialize the model on each tool call.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.