Add Embedding, Reranker (and optional OCR) model fields to Agent configuration for building RAG pipelines at the agent level
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 49.2k
- Forks
- 4.5k
- Avg merge
- 1d 3m
- Merged PRs (30d)
- 239
Description
Description:
Currently, the ModelSettings section of an agent's configuration exposes Model, MultimodalModel, TranscriptionModel, TranscriptionLanguage, TTSModel, and PlanReviewerModel.
However, there is no dedicated field for:
- An Embedding Model override at the per-agent level (the only embedding model configuration today is the pool-wide LOCALAI_AGENT_POOL_EMBEDDING_MODEL / agent_pool_embedding_model setting used for the Knowledge Base, not something selectable per agent) .
- A Reranker Model to run a second-stage relevance-scoring pass (via /v1/rerank) on Knowledge Base search results before they reach the LLM.
- An optional OCR Model for pre-processing scanned documents/images before chunking and embedding.
Requested feature:
Extend AgentConfig (and the corresponding ModelSettings UI fields in configmeta.go) with new optional fields, e.g.:
- embedding_model (string) — overrides the pool default EmbeddingModel for this agent's Knowledge Base ingestion/search.
- reranker_model (string) — if set, automatically invokes /v1/rerank on the top-K KB search results before passing them to the agent's LLM context, refining relevance ordering.
- ocr_model (string, optional) — if set, runs OCR/vision-to-text extraction on uploaded scanned documents/images before chunking, so that Knowledge Base ingestion for Enable Knowledge Base / kb_mode configmeta.go:91-101 can handle scanned PDFs and images properly.
Motivation:
Today, building an embed → rerank → LLM pipeline for RAG requires external orchestration outside of the agent configuration, since Knowledge Base collections endpoints (/api/agents/collections/:name/search, /upload, etc.) have no built-in reranking or OCR hook. Making these configurable per-agent would let users build fully self-contained multi-stage RAG pipelines natively, without needing a separate script/action to call /v1/rerank manually.
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading AgentConfig and the ModelSettings fields in configmeta.go, including the Knowledge Base settings around configmeta.go:91-101. Trace the collection search, upload, and /v1/rerank entry points to understand where per-agent embedding, reranking, and OCR settings would connect. Done means the requested optional fields are configurable per agent and the described Knowledge Base pipeline behavior is implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai, backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100