kubeflow / kubeflow/docs-agent
Performance: avoid reloading embedding model on every query in docs-agent API
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
Summary
server-https/app.py (and server/app.py) instantiate SentenceTransformer inside milvus_search() for every tool call. This causes avoidable latency and memory churn.
Problem
Current behavior:
- Each request that triggers
search_kubeflow_docsre-creates the embedding model. - Model load time dominates query latency under repeated traffic.
Expected behavior:
- Embedding model should be initialized once per process and reused across queries. for low latency
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing server-https/app.py and server/app.py, focusing on milvus_search() and the search_kubeflow_docs entry point where SentenceTransformer is instantiated. Confirm the current per-call behavior, then verify that the embedding model is initialized once per process and reused across repeated queries without changing search results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100