feat: Serve embedding model as a dedicated KServe InferenceService instead of per-run install

Problem

The current chunk_and_embed pipeline component installs sentence-transformers
and torch from scratch on every single pipeline run:

@dsl.component(
    base_image="pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime",
    packages_to_install=["sentence-transformers", "langchain"]
)
def chunk_and_embed(
    github_data: dsl.Input[dsl.Dataset],
    repo_name: str,
    ...
):

This causes:

• 5–10 minutes of install overhead per pipeline run before any actual work starts
• Heavy base image (pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime is ~8GB) used just for embeddings
• Model re-downloaded from HuggingFace on every run
• No reuse — the API server (server/app.py, server-https/app.py) also loads embeddings separately, meaning the same model is loaded in two completely different places

This is already acknowledged in the README under "Future Improvements":

"A better improvement would be using the embedding model as a service where users
could call the service instead of installing heavy sentence transformers package every time.
This would reduce pipeline execution time, lower resource requirements, enable better
caching and optimization, improve scalability."

---

Proposed Solution

Deploy sentence-transformers/all-mpnet-base-v2 as a dedicated KServe InferenceService,
and update the pipeline component to call it via HTTP.

1. New file: manifests/embedding-inference-service.yaml

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: embedding-model
  namespace: ${NAMESPACE}
spec:
  predictor:
    minReplicas: 1
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=all-mpnet-base-v2
        - --model_id=sentence-transformers/all-mpnet-base-v2
        - --task=feature-extraction
      resources:
        requests:
          cpu: "2"
          memory: "4Gi"
        limits:
          cpu: "4"
          memory: "8Gi"

2. Update pipelines/kubeflow-pipeline.py

Replace the heavy pytorch base image with a lightweight HTTP client:

# Before
@dsl.component(
    base_image="pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime",
    packages_to_install=["sentence-transformers", "langchain"]
)
def chunk_and_embed(...):
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer('all-mpnet-base-v2')
    embeddings = model.encode(chunks)

# After
@dsl.component(
    base_image="python:3.9",
    packages_to_install=["requests", "langchain-text-splitters"]
)
def chunk_and_embed(...):
    import requests
    EMBEDDING_SERVICE_URL = os.getenv(
        "EMBEDDING_SERVICE_URL",
        "http://embedding-model.${NAMESPACE}.svc.cluster.local/v1/models/all-mpnet-base-v2:predict"
    )
    response = requests.post(EMBEDDING_SERVICE_URL, json={"instances": chunks})
    embeddings = response.json()["predictions"]

3. Add EMBEDDING_SERVICE_URL to environment variables table in README.md

---

Benefits

	Before	After
Base image size	~8GB (pytorch)	~200MB (python:3.9)
Install time per run	5–10 min	~30 sec
Model reuse	No	Yes (shared service)
Reusable by API server	No	Yes
Consistent embeddings	Risk of drift	Guaranteed

---

Files Changed

File	Change
manifests/embedding-inference-service.yaml	New file — KServe InferenceService for embedding model
pipelines/kubeflow-pipeline.py	Update chunk_and_embed to call embedding service via HTTP
README.md	Add EMBEDDING_SERVICE_URL to env vars table, update architecture description

---

I'd like to work on this if maintainers are aligned on the approach.

Start with pipelines/kubeflow-pipeline.py and compare the current chunk_and_embed component with the embedding-loading paths in server/app.py and server-https/app.py. Review the proposed KServe manifest in manifests/embedding-inference-service.yaml and the README environment-variable table. Done means the service definition, HTTP-based pipeline integration, and EMBEDDING_SERVICE_URL documentation are updated consistently.