mudler / mudler/LocalAI

Add Embedding, Reranker (and optional OCR) model fields to Agent configuration for building RAG pipelines at the agent level

Open
#11,753 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/api area/webui enhancement
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

Description:

Currently, the ModelSettings section of an agent's configuration exposes Model, MultimodalModel, TranscriptionModel, TranscriptionLanguage, TTSModel, and PlanReviewerModel.

However, there is no dedicated field for:

  • An Embedding Model override at the per-agent level (the only embedding model configuration today is the pool-wide LOCALAI_AGENT_POOL_EMBEDDING_MODEL / agent_pool_embedding_model setting used for the Knowledge Base, not something selectable per agent) .
  • A Reranker Model to run a second-stage relevance-scoring pass (via /v1/rerank) on Knowledge Base search results before they reach the LLM.
  • An optional OCR Model for pre-processing scanned documents/images before chunking and embedding.

Requested feature:

Extend AgentConfig (and the corresponding ModelSettings UI fields in configmeta.go) with new optional fields, e.g.:

  • embedding_model (string) — overrides the pool default EmbeddingModel for this agent's Knowledge Base ingestion/search.
  • reranker_model (string) — if set, automatically invokes /v1/rerank on the top-K KB search results before passing them to the agent's LLM context, refining relevance ordering.
  • ocr_model (string, optional) — if set, runs OCR/vision-to-text extraction on uploaded scanned documents/images before chunking, so that Knowledge Base ingestion for Enable Knowledge Base / kb_mode configmeta.go:91-101 can handle scanned PDFs and images properly.

Motivation:

Today, building an embed → rerank → LLM pipeline for RAG requires external orchestration outside of the agent configuration, since Knowledge Base collections endpoints (/api/agents/collections/:name/search, /upload, etc.) have no built-in reranking or OCR hook. Making these configurable per-agent would let users build fully self-contained multi-stage RAG pipelines natively, without needing a separate script/action to call /v1/rerank manually.

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading AgentConfig and the ModelSettings fields in configmeta.go, including the Knowledge Base settings around configmeta.go:91-101. Trace the collection search, upload, and /v1/rerank entry points to understand where per-agent embedding, reranking, and OCR settings would connect. Done means the requested optional fields are configurable per agent and the described Knowledge Base pipeline behavior is implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai, backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.