kubeflow / kubeflow/docs-agent

SentenceTransformer model reloaded on every search request

Open Beginner friendly
#189 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

## Problem

The `milvus_search()` function in both `server/app.py` (line 77) and `server-https/app.py` (line 123) initializes a new `SentenceTransformer` instance on every call:
```python
encoder = SentenceTransformer(EMBEDDING_MODEL)
```

Loading `sentence-transformers/all-mpnet-base-v2` takes ~2-5 seconds and ~400MB of memory each time. In agentic RAG workflows with multiple tool calls per user turn, this adds 10-25 seconds of unnecessary latency per query.

## Proposed Fix

Initialize the model once at module level and reuse it across all requests.

Contributor guide

Open the contributing guide

Research direction

Read milvus_search() in server/app.py around line 77 and server-https/app.py around line 123, starting with how each entry point initializes the encoder. Verify the search request path, then confirm both entry points reuse one model instance rather than loading it for every request.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.