kubeflow / kubeflow/docs-agent
Performance Bug: SentenceTransformer is re-initialized on every Milvus search request
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
### Bug Description
In `server/app.py`, the `SentenceTransformer(EMBEDDING_MODEL)` is instantiated inside the `milvus_search()` function. This causes the application to load the entire embedding model into memory on every single tool call, resulting in severe latency spikes (multiple seconds per search) and risking OOM crashes under concurrent WebSocket connections.
### Proposed Solution
Move the `SentenceTransformer` initialization to the global scope so it acts as a singleton loaded once at server startup. The `milvus_search` function should reference this global instance.
I have tested this locally and it drastically reduces tool execution latency. I will open a PR with the fix shortly.
Contributor guide
Research direction
Inspect server/app.py, starting with milvus_search() and the SentenceTransformer(EMBEDDING_MODEL) construction. Move initialization to global scope, have milvus_search() use that instance, and verify searches no longer re-initialize the model on each tool call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100