kubeflow / kubeflow/docs-agent
perf(server): SentenceTransformer model re-loaded on every search request (no caching)
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
## Problem
In both `server/app.py` (line 77) and `server-https/app.py` (line 123), the `SentenceTransformer` model is instantiated inside the request handler, meaning it is fully re-loaded from disk on **every** search request:
```python
# Called on every request:
encoder = SentenceTransformer(EMBEDDING_MODEL)
query_vec = encoder.encode(query).tolist()
```
The model weights are static and do not change between requests. Re-loading them on every call adds ~4.2s of unnecessary latency per request and wastes memory.
## Benchmark
I profiled the before/after behaviour locally:
| | Avg per call | Total (5 calls) |
|---|---|---|
| Before (re-load every call) | 4.220s | 21.102s |
| After (load once at startup) | 0.084s | 0.422s |
| **Speedup** | **50x faster** | **20.68s saved** |
The one-time startup cost is 4.28s — paid once when the server starts, not on every request.
## Fix
Move `SentenceTransformer(EMBEDDING_MODEL)` to module-level initialization so it is loaded once at startup:
```python
# At module level (outside any function):
encoder = SentenceTransformer(EMBEDDING_MODEL)
# Inside the request handler:
query_vec = encoder.encode(query).tolist()
```
## Files
- `server/app.py` line 77
- `server-https/app.py` line 123
Benchmark files have been attached below :
[benchmark_model_load.py](https://github.com/user-attachments/files/25795392/benchmark_model_load.py)
[benchmark_results.txt](https://github.com/user-attachments/files/25795399/benchmark_results.txt)
Contributor guide
Research direction
Start with the request handlers at server/app.py line 77 and server-https/app.py line 123, then review benchmark_model_load.py and benchmark_results.txt. Confirm the model-loading behavior in both paths and verify that repeated searches match the benchmark's reduced latency after the change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning, performance
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100