kubeflow / kubeflow/docs-agent

perf(server): SentenceTransformer model re-loaded on every search request (no caching)

Open
#115 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

## Problem

In both `server/app.py` (line 77) and `server-https/app.py` (line 123), the `SentenceTransformer` model is instantiated inside the request handler, meaning it is fully re-loaded from disk on **every** search request:
```python
# Called on every request:
encoder = SentenceTransformer(EMBEDDING_MODEL)
query_vec = encoder.encode(query).tolist()
```

The model weights are static and do not change between requests. Re-loading them on every call adds ~4.2s of unnecessary latency per request and wastes memory.

## Benchmark

I profiled the before/after behaviour locally:

| | Avg per call | Total (5 calls) |
|---|---|---|
| Before (re-load every call) | 4.220s | 21.102s |
| After (load once at startup) | 0.084s | 0.422s |
| **Speedup** | **50x faster** | **20.68s saved** |

The one-time startup cost is 4.28s — paid once when the server starts, not on every request.

## Fix

Move `SentenceTransformer(EMBEDDING_MODEL)` to module-level initialization so it is loaded once at startup:
```python
# At module level (outside any function):
encoder = SentenceTransformer(EMBEDDING_MODEL)

# Inside the request handler:
query_vec = encoder.encode(query).tolist()
```

## Files
- `server/app.py` line 77
- `server-https/app.py` line 123

Benchmark files have been attached below :

[benchmark_model_load.py](https://github.com/user-attachments/files/25795392/benchmark_model_load.py)

[benchmark_results.txt](https://github.com/user-attachments/files/25795399/benchmark_results.txt)

Contributor guide

Open the contributing guide

Research direction

Start with the request handlers at server/app.py line 77 and server-https/app.py line 123, then review benchmark_model_load.py and benchmark_results.txt. Confirm the model-loading behavior in both paths and verify that repeated searches match the benchmark's reduced latency after the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning, performance
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.