Proposal: Add Advanced Retrieval Processors (CommunityToolkit.DataRetrieval)

# Proposal: Add Advanced Retrieval Processors (`CommunityToolkit.DataRetrieval`)

## Summary

This issue proposes adding advanced retrieval pipeline processors to the AI Community Toolkit, building on the `Microsoft.Extensions.DataRetrieval` abstractions (proposed in dotnet/extensions#7507). These processors enable sophisticated RAG (Retrieval-Augmented Generation) patterns — query expansion, hypothetical document embeddings, LLM reranking, corrective RAG, and more — through a fluent DI-friendly builder API.

## Motivation

### What This Package Provides

The `Microsoft.Extensions.DataRetrieval` packages (proposed in dotnet/extensions) give you the **abstractions** — the pipeline structure, base classes, and data types. But abstractions alone don't help you build a better RAG system. You still need **concrete implementations** of the techniques that actually improve retrieval quality.

That's what this package provides: production-ready, tested implementations of proven retrieval techniques, wired up through a fluent builder that handles DI registration automatically.

### The Techniques Explained

#### Pre-Search: Making Your Query Better Before Searching

**Multi-Query Expansion** (`MultiQueryExpander`)

The problem: A single query embedding may not capture all aspects of what the user is asking. "How do I handle errors in my API?" is one question, but it relates to exception handling, middleware, HTTP status codes, and validation — each might be stored in different chunks.

The solution: Ask an LLM to generate N variant phrasings of the original query. Search with ALL of them, then merge results using Reciprocal Rank Fusion (RRF) — a technique that boosts chunks appearing in multiple result sets. A chunk that appears for 3 out of 5 query variants is almost certainly relevant.

**HyDE — Hypothetical Document Embeddings** (`HydeQueryTransformer`)

The problem: Users ask short questions ("How do I configure CORS?") but documents contain long explanations. Short questions and long answers live in different regions of vector space — their embeddings aren't as close as you'd expect.

The solution: Ask an LLM to generate a *hypothetical answer* to the question (it doesn't need to be correct). Then search using the hypothetical answer's embedding instead. Since a hypothetical answer looks like a document chunk, its embedding is naturally closer to the real answer chunks in vector space.

**Adaptive Routing** (`AdaptiveRouter`)

The problem: Not all queries benefit from the same retrieval strategy. A simple factual question ("What's the default timeout?") needs basic vector search. A broad architectural question ("How does the authentication system work?") needs hierarchical tree search. A question about a specific person or technology might benefit from metadata-filtered search.

The solution: Ask an LLM to classify the query's complexity and type, then route to the optimal search paradigm. Simple → standard vector search. Thematic/broad → tree search through summaries. Entity-specific → filtered vector search.

#### Post-Search: Improving Results After Search

**LLM Reranking** (`LlmReranker`)

The problem: Vector similarity is an approximation. The "closest" vectors aren't always the most *relevant* for answering the specific question. A chunk about "configuring CORS headers" might be geometrically closer than one about "CORS middleware setup," even though the latter actually answers the question better.

The solution: Retrieve more candidates than you need (say 20), then ask an LLM to rank them by actual relevance to the question. The LLM reads each chunk and the query, then orders them by how well each chunk answers the question. Return only the top N.

**Corrective RAG** (`CragValidator`)

The problem: Sometimes vector search returns chunks that are topically related but don't actually answer the question. If you pass these to the generation LLM, it may hallucinate an answer that sounds plausible but isn't grounded in the retrieved content.

The solution: Before using the results, ask an LLM to score retrieval quality (1-5). Based on the score, route through three paths:
- **Correct** (4-5): Results are relevant, proceed normally
- **Ambiguous** (2-3): Results are partially relevant, attempt refinement
- **Incorrect** (1): Results are irrelevant, fall back to web search or acknowledge uncertainty

#### Advanced Orchestrators: Multi-Step Patterns

**Self-RAG** (`SelfRagOrchestrator`)

A self-reflective loop: retrieve → generate a draft answer → evaluate whether the draft is well-grounded in the retrieved chunks → if not, refine the query and try again. This catches hallucination before it reaches the user.

**Speculative RAG** (`SpeculativeRagOrchestrator`)

A parallel drafting approach: generate multiple candidate answers from different chunk subsets simultaneously, then use an LLM to select the best one. Trades compute for quality — useful when accuracy matters more than latency.

### Why a Fluent Builder?

Instead of manually wiring processors and DI registrations:

```csharp
// Without builder (manual, error-prone)
services.AddSingleton(sp => {
var pipeline = new RetrievalPipeline();
pipeline.QueryProcessors.Add(new MultiQueryExpander(sp.GetRequiredService()));
pipeline.ResultProcessors.Add(new LlmReranker(sp.GetRequiredService()));
return pipeline;
});

// With builder (discoverable, concise)
services.AddRetrievalPipeline()
.UseQueryExpansion()
.UseLlmReranking()
.AsRetriever(
sp => sp.GetRequiredService>(),
record => record.Content);
```

The builder is discoverable via IntelliSense, handles DI resolution, and the `AsRetriever()` terminal registers a ready-to-inject `IRetriever` singleton.

## Proposed Packages

### Package: `CommunityToolkit.DataRetrieval`

LLM-powered processors + fluent builder for composing retrieval pipelines via dependency injection.

#### Fluent Builder API

```csharp
using CommunityToolkit.DataRetrieval;

builder.Services.AddRetrievalPipeline()
.UseQueryExpansion(o => o.VariantCount = 5)
.UseHyDE()
.UseLlmReranking(o => o.MaxResults = 5)
.UseCrag()
.AsRetriever(
sp => sp.GetRequiredService>(),
record => record.Content);
```

#### Query Processors (pre-search)

| Type | Technique | What It Does |
|------|-----------|-------------|
| `MultiQueryExpander` | Multi-Query Expansion | Generates N query variants via LLM, searches with all of them, then merges results using Reciprocal Rank Fusion — chunks appearing across multiple variant results are boosted to the top |
| `HydeQueryTransformer` | Hypothetical Document Embeddings | Generates a hypothetical answer to the query, then searches using that answer's embedding — bridges the "short question ↔ long document" vector space gap |
| `AdaptiveRouter` | Adaptive Routing | Classifies query complexity (simple/thematic/entity-specific) and routes to the optimal search strategy — not all questions benefit from the same retrieval approach |
| `TreeSearchRetriever` | RAPTOR Tree Search | Top-down hierarchical traversal starting from corpus-level summaries down to leaf chunks — ideal for broad or thematic questions where you need context at multiple abstraction levels |

#### Result Processors (post-search)

| Type | Technique | What It Does |
|------|-----------|-------------|
| `LlmReranker` | LLM Reranking | Retrieves extra candidates, then asks an LLM to rank them by actual relevance (not just vector distance) — returns only the top N most relevant passages |
| `CragValidator` | Corrective RAG | Scores retrieval quality 1-5 and routes through correct/ambiguous/incorrect paths — catches cases where retrieved chunks are topically related but don't actually answer the question |

#### Orchestrators (advanced multi-step)

| Type | Technique | What It Does |
|------|-----------|-------------|
| `SelfRagOrchestrator` | Self-RAG | Self-reflective loop: retrieve → generate draft → evaluate groundedness → refine if needed — catches hallucination before it reaches the user |
| `SpeculativeRagOrchestrator` | Speculative RAG | Generates multiple candidate answers from different chunk subsets in parallel, then selects the best one — trades compute for answer quality |

#### DI Extensions

```csharp
public static class DataRetrievalServiceCollectionExtensions
{
// No-arg: creates default pipeline, returns builder for composition
public static RetrievalPipelineBuilder AddRetrievalPipeline(
this IServiceCollection services);

// Factory overload: bring your own pipeline instance
public static RetrievalPipelineBuilder AddRetrievalPipeline(
this IServiceCollection services,
Func pipelineFactory);
}
```

#### Builder Terminal: `AsRetriever`

```csharp
// Registers IRetriever singleton binding pipeline → collection
builder.Services.AddRetrievalPipeline()
.UseQueryExpansion()
.UseLlmReranking()
.AsRetriever(
sp => sp.GetRequiredService>(),
record => record.Content);

// Then inject IRetriever anywhere:
public class ChatService(IRetriever retriever)
{
public async Task AnswerAsync(string question)
{
var results = await retriever.RetrieveAsync(question, topK: 5);
// ... use results.Chunks for grounding
}
}
```

### Package: `CommunityToolkit.DataRetrieval.OnnxReranker`

Local ONNX cross-encoder reranker for zero-cloud-dependency reranking.

```csharp
using CommunityToolkit.DataRetrieval.OnnxReranker;

// Use with pipeline directly
pipeline.ResultProcessors.Add(new CrossEncoderReranker(new CrossEncoderRerankerOptions
{
ModelPath = "models/ms-marco-MiniLM-L-6-v2.onnx",
TokenizerPath = "models/tokenizer.json",
MaxResults = 5
}));

// Or as IReranker for IReranker-consuming APIs
IReranker reranker = new CrossEncoderReranker(options);
```

**Features:**
- Supports ms-marco-MiniLM, BGE-reranker, and any ONNX cross-encoder model
- Text-pair tokenization → ONNX inference → sigmoid normalization
- Lazy initialization (model loaded on first use)
- `IDisposable` for native ONNX resource cleanup
- Implements both `RetrievalResultProcessor` AND `IReranker` (dual interface)

**Dependencies:** `Microsoft.ML.OnnxRuntime.Managed` + `Microsoft.ML.Tokenizers`

## Relationship to Upstream

| Package | Layer | Description |
|---------|-------|-------------|
| `Microsoft.Extensions.DataRetrieval.Abstractions` | Contracts | `RetrievalQuery`, `RetrievalChunk`, processor base classes, `IRetriever` |
| `Microsoft.Extensions.DataRetrieval` | Pipeline | `RetrievalPipeline`, `VectorStoreRetriever`, `AsRetriever()` |
| **`CommunityToolkit.DataRetrieval`** | **Implementations** | **Concrete processors + builder + DI** |
| **`CommunityToolkit.DataRetrieval.OnnxReranker`** | **Local ML** | **ONNX cross-encoder reranking** |

This follows the same layering as document processing:
- `Microsoft.Extensions.DataIngestion` (abstractions in dotnet/extensions)
- `CommunityToolkit.DocumentProcessing.PdfPig.*` (implementations in CommunityToolkit)

## Design Decisions

| Decision | Rationale |
|----------|-----------|
| Fluent builder (not manual registration) | Discoverability via IntelliSense; reduces DI boilerplate; matches ASP.NET patterns (`AddAuthentication().AddJwtBearer()`) |
| All processors use `IChatClient` | Works with any M.E.AI provider (OpenAI, Azure, Ollama, etc.) |
| `AsRetriever()` as terminal | Natural pipeline → endpoint transition; registers `IRetriever` in DI for downstream injection |
| ONNX reranker in separate package | Heavy native dependencies (OnnxRuntime); not everyone needs local ML |
| Options pattern for configuration | `Action?` on each builder method; familiar .NET pattern |
| Orchestrators as standalone classes | Self-RAG/Speculative RAG are full orchestration patterns, not single-stage processors |

## Dependencies

**`CommunityToolkit.DataRetrieval`:**
- `Microsoft.Extensions.DataRetrieval` (≥ 10.5.0-preview)
- `Microsoft.Extensions.DataRetrieval.Abstractions`
- `Microsoft.Extensions.AI.Abstractions`
- `Microsoft.Extensions.DependencyInjection.Abstractions`
- `Microsoft.Extensions.VectorData.Abstractions`
- `Microsoft.Extensions.Logging.Abstractions`

**`CommunityToolkit.DataRetrieval.OnnxReranker`:**
- All of the above, plus:
- `Microsoft.ML.OnnxRuntime.Managed`
- `Microsoft.ML.Tokenizers`

## Implementation

A reference implementation exists on the [`feature/data-retrieval`](https://github.com/luisquintanilla/AI/tree/feature/data-retrieval) branch including:
- Both packages with full XML documentation
- Fluent builder with IntelliSense-friendly method signatures
- All processors tested against real retrieval scenarios in the advanced-rag reference application