Search/RAG: adopt age-decay + score-aware fusion, source-aware chunking, and continuous ingestion (lessons from Cerebras Knowledge)

## Summary

Cerebras published [*How we built our knowledge base*](https://www.cerebras.ai/blog/how-we-built-our-knowledge-base) describing an internal RAG system answering ~15K questions/day over Slack, GitHub, Confluence, Google Docs, and Jira. Their write-up crystallizes a few retrieval-layer techniques that MJ's search/RAG stack does **not** do today. This issue captures those gaps as concrete, MJ-scoped enhancements.

> The original blog is currently behind a Cloudflare 403 for automated fetches; syndicated breakdowns with the same technical detail: [Dealroom](https://app.dealroom.co/news/note/inside-cerebras-knowledge-how-cerebras-built-its-enterprise-knowledge-base), [MindStudio](https://www.mindstudio.ai/blog/enterprise-rag-knowledge-base-cerebras-how-it-works).

### What Cerebras does (the relevant parts)
1. **One Postgres table** as the substrate — every source normalizes into `embedding + raw text + summary + metadata`, queried behind a single access layer (authn/authz/audit/analytics).
2. **Extract from where data is generated** (Slack/GitHub/Docs/Jira connectors) rather than migrating to a "source of truth."
3. **Hybrid retrieval fusing four signals**, because vector search alone was insufficient: full-text (exact tokens — error strings, flags, hostnames), embeddings (paraphrase), **IDF** (signal vs. filler), and **age decay** (stale answers rank lower) — then **reranking**.
4. **Source-aware chunking** — code uses the open-source [CocoIndex](https://github.com/cocoindex-io/cocoindex) framework to keep embeddings of 40GB+ repos current, splitting along **language-aware boundaries, coarse→fine**, incrementally re-embedding only what changed.
5. **Their thesis:** the *retrieval* layer (chunking + embedding + hybrid + rerank) drives answer quality far more than the generation-model choice.

## Where MJ already matches this

MJ is closer to this architecture than it might seem — most pillars already exist as primitives:

| Cerebras pillar | MJ equivalent |
|---|---|
| Pluggable source connectors | `BaseSearchProvider` (vector/fulltext/entity/storage) + external providers (Elasticsearch/Typesense/Azure AI/OpenSearch) under `packages/SearchEngine/src/providers/` |
| Hybrid fusion | `SearchEngine.Search` + `SearchFusion` + `ComputeRRF` |
| FTS + embedding blend | `SearchEntity` hybrid mode; cross-source in `SearchEngine` |
| Reranking | `BaseReRanker` catalog — Cohere, Voyage, OpenAI, BGE, Noop + budget guard (`packages/SearchEngine/src/rerankers/`) |
| Access layer | Permission push-down (§3.6), `SearchScopePermissionResolver`, `SearchExecutionLog` |
| Agent retrieval | Search Scopes + pre-execution RAG (`packages/AI/Agents/src/agent-pre-execution-rag.ts`) + `__Scoped_Search` |

Reference docs: [`guides/SEARCH_OVERVIEW_GUIDE.md`](guides/SEARCH_OVERVIEW_GUIDE.md), [`guides/SEARCH_SCOPES_AND_RAG_GUIDE.md`](guides/SEARCH_SCOPES_AND_RAG_GUIDE.md), [`guides/ENTITY_SEARCH_GUIDE.md`](guides/ENTITY_SEARCH_GUIDE.md).

## The gaps — proposed work (ranked by leverage)

### 1. Age decay in fusion — *highest leverage, lowest cost*
MJ's RRF is **purely rank-based** and has **no recency signal**. `ComputeRRF` computes `w_i / (k + rank_i(d))` and discards any temporal dimension — see [`packages/MJCore/src/generic/scoring/ReciprocalRankFusion.ts:53-98`](packages/MJCore/src/generic/scoring/ReciprocalRankFusion.ts#L53-L98). `SearchFusion.Fuse` / `CrossScopeFusion` have no decay stage — see [`packages/SearchEngine/src/generic/SearchFusion.ts`](packages/SearchEngine/src/generic/SearchFusion.ts). A 2-year-old doc and a fresh one rank identically on retrieval merit alone.

**Proposal:** add a `recencyDecay` block to `SearchScope.ScopeConfig` (half-life + per-source timestamp field), applied as a multiplier on the fused score in `SearchFusion` before the final sort. Honor a per-agent override alongside the existing `FusionWeightsOverride` path documented in `SEARCH_SCOPES_AND_RAG_GUIDE.md` §3. Contained change; disproportionately helps any corpus that goes stale (Slack, incident logs, rotating policies).

### 2. Preserve native provider scores (IDF / BM25) instead of collapsing to rank
SQL Server / PG full-text already compute BM25-ish relevance, but MJ **throws that score away** when it converts each provider list to *ranks* for RRF (`ReciprocalRankFusion.ts:75-88`). Cerebras explicitly keeps IDF to down-weight filler. RRF is score-scale independent by design (a virtue for incomparable sources), but for the FTS list specifically we're discarding usable saliency.

**Proposal:** carry each provider's native relevance score through `SearchResultItem` and offer an opt-in **score-based** (rather than rank-based) blend for lexical providers in `SearchFusion`, so IDF/BM25 signal survives fusion. Default behavior unchanged.

### 3. Source-aware / code-aware chunking — *biggest conceptual gap*
MJ's `EntityDocument` model embeds **one vector per record** from a Nunjucks template (see `ENTITY_SEARCH_GUIDE.md` and `metadata/entity-documents/`). Great for structured catalog rows; **wrong for long documents and code** — a 40-page policy or a source file becomes one blurry embedding. The scopes design already references chunk-shaped hooks (`getMatchingChunks`, chunk-shaped query rewrites in `SEARCH_SCOPES_AND_RAG_GUIDE.md`) but there is **no first-class chunking pipeline**.

**Proposal:** design a chunk table (e.g. `EntityRecordDocumentChunk`) with parent-record linkage, a pluggable chunker (recursive / semantic / code-AST — CocoIndex's coarse→fine, language-aware approach as the reference), and **incremental** re-embedding keyed on change, reusing the dirty-field `Save()` embedding contract already documented in `SEARCH_SCOPES_AND_RAG_GUIDE.md` §18 ("Embedding regeneration contract"). This deserves its own design doc, on par with the Search Scopes plan.

### 4. Continuous ingestion vs. our daily cron
Cerebras keeps embeddings *current*. MJ's default is a 4am daily `Entity Vector Sync` job, plus event-driven regen on only the three core entities that carry `EmbeddingVector` columns (`MJ: Queries`, `MJ: AI Agent Notes`, `MJ: AI Agent Examples` — see the regeneration contract in `SEARCH_SCOPES_AND_RAG_GUIDE.md` §18). For a knowledge base expected to reflect "what someone said in Slack an hour ago," daily is too coarse.

**Proposal:** extend the `Save()`-hook embedding contract to the ingestion path so new content embeds on arrival; reserve the cron for backfill/repair. Track staleness explicitly (the guide already floats an `EmbeddingRegeneratedAt` column + maintenance action — worth building now).

## Design tension worth recording
Cerebras collapsed everything into **one Postgres table**; MJ is deliberately **federated** (many providers/indexes, RRF over them). Theirs buys simplicity + a single access chokepoint; ours buys pluggability + multi-tenant scope isolation. We should **not** abandon MJ's model — but their single-access-layer discipline reinforces our existing rule that **every provider must push permissions down before fusion** (`SEARCH_SCOPES_AND_RAG_GUIDE.md` §3.6/§4; watch `lateFilteredCount` telemetry). That's the part of "one table, one access layer" to keep religiously even without one table.

## Suggested sequencing
- **Now:** #1 (age decay) and #2 (score-aware fusion) — days of work in `SearchFusion` / `ReciprocalRankFusion`, immediate quality wins, no schema churn.
- **Next (own design doc):** #3 chunking — touches `EntityDocument` + vector-sync pipeline.
- **Alongside #3:** #4 continuous ingestion + staleness tracking.

## References
- Cerebras, *How we built our knowledge base* — https://www.cerebras.ai/blog/how-we-built-our-knowledge-base
- CocoIndex — https://github.com/cocoindex-io/cocoindex
- MJ code: `packages/MJCore/src/generic/scoring/ReciprocalRankFusion.ts`, `packages/SearchEngine/src/generic/SearchFusion.ts`, `packages/SearchEngine/src/generic/SearchEngine.ts`, `packages/AI/Agents/src/agent-pre-execution-rag.ts`
- MJ docs: `guides/SEARCH_OVERVIEW_GUIDE.md`, `guides/SEARCH_SCOPES_AND_RAG_GUIDE.md`, `guides/ENTITY_SEARCH_GUIDE.md`

Start with packages/MJCore/src/generic/scoring/ReciprocalRankFusion.ts, packages/SearchEngine/src/generic/SearchFusion.ts, and the three search guides to understand the existing fusion and embedding contracts. The issue spans age decay, score-aware fusion, chunking, and ingestion without a single test or scoped entry point; done requires a maintainer-approved design and narrower implementation plan.

Written by the indexing model from the issue text.