kubeflow / kubeflow/docs-agent
feat: Add centralized embedding-service/ to eliminate per-component model loading (ADR-004)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
Problem
sentence-transformers + torch are installed independently across 5 components. The README explicitly flags this under "Future Improvements".
| Component | Current | Impact |
|---|---|---|
pipelines/*.py |
pytorch:2.3.0-cuda base |
~8 GB image, 5–10 min install per run |
server/app.py, server-https/app.py |
SentenceTransformer per-process |
~2–3 GB image bloat |
kagent-feast-mcp/mcp-server/server.py |
SentenceTransformer local |
Same bloat, no sharing |
Solution
Deploy all-mpnet-base-v2 as a standalone FastAPI microservice with model weights baked at build time. All 5 components call POST /embed via HTTP.
Why standalone FastAPI over KServe InferenceService (#48):
- No KServe cluster dependency — works on any K8s, Docker Compose, or local
- Zero cold-start — model baked into image, no HuggingFace download at runtime
- Simpler: one container, one endpoint
Files
| File | Change |
|---|---|
embedding-service/app.py |
New — FastAPI /embed, model loaded once at startup |
embedding-service/Dockerfile |
New — model weights baked at build time |
manifests/embedding-service/ |
New — K8s Deployment + ClusterIP Service |
pipelines/*.py (2 files) |
pytorch:2.3.0 → python:3.9; batch-64 HTTP embedding |
server/app.py, server-https/app.py |
Remove SentenceTransformer; httpx.AsyncClient embedding call |
kagent-feast-mcp/mcp-server/server.py |
HTTP embedding call |
*/requirements.txt (5 files) |
Remove sentence-transformers + torch |
Impact
| Metric | Before | After |
|---|---|---|
| Pipeline base image | ~8 GB | ~50 MB |
| Per-run install time | 5–10 min | ~30 sec |
| Model cold-start | HuggingFace download per run | Zero |
Components with torch dep |
5 | 0 |
Related
- #48 — KServe approach (superseded by this, more portable)
- #63 — per-query model reload (resolved)
- #72 — parent pre-work tracking
- #59 — GSoC design discussion
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the listed embedding-service files, manifests/embedding-service/, the two pipeline files, server/app.py, server-https/app.py, and kagent-feast-mcp/mcp-server/server.py. Trace each current model-loading path and the five requirements files before implementing the centralized service and HTTP callers. Done means the service exposes POST /embed, all five components use it, and sentence-transformers and torch are removed from their requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, fastapi, kubernetes, python, pytorch
- Domain
- api, backend, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100