Example: deterministic RAG eval — retrieval + ACL scored reproducibly, offline, in CI

An example that uses Backlot as an evaluation substrate rather than a thing to read from: retrieve over a served corpus, score the retrieval against a gold set the corpus itself declares, and get the same numbers every run — offline, with no API key, as a job that can block a PR.

## Why

Every consumer example so far — `using-official-sdk`, `using-llamaindex-readers`, `using-mcp-with-agents`, `using-mirage` — shows a client **reading** the mock. They prove the surfaces work. None of them uses the property that makes this server different from a recorded fixture: what it serves is *derived*, so a query's answer set is the same on the first run, the tenth, page three, and a fresh import of the same corpus.

That is the missing piece for retrieval evaluation. A RAG eval against live SaaS measures the vendor's index and last week's data drift along with your retriever; against hand-written fixtures it measures whether your fixtures agree with your assumptions. Here the gold set is a fact about the corpus you loaded, so a score that moves means the retriever moved.

## Scope

**In:** retrieval scored against known-relevant documents, with non-judge metrics, reproducibly, in CI.

**Out**, deliberately — each of these is a separate topic and each one would blur the claim this example exists to make:

- LLM-judge scoring (answer faithfulness, relevance-as-graded-by-a-model). Not reproducible in the sense this example is selling; see the caveat below.
- Tracing and observability. Diffing spans across two runs is a different demonstration with a different dependency footprint.
- A tour of eval frameworks. One path, no framework required to run it.
- Grading the public dataset (#19). That harness runs over a downloaded multi-source corpus and is its own deliverable in its own repo; this example runs on a corpus small enough to read in one sitting, with nothing to fetch.

## What to build

`examples/deterministic-rag-eval/`:

1. **A corpus that carries its own answer key.** A small BYO-JSONL corpus written for the eval, passed to `backlot.mock_server(records=...)`, with a companion file of queries → the documents that answer them. It ships its own corpus rather than leaning on the bundled one on purpose: the bundled corpus is a demo, its contents are free to change, and an eval that hardcodes ids out of it is asserting on something no one promised to keep.
2. **A retrieval step through a real client.** Crawl a couple of sources with an official SDK or a reader, index locally (BM25 or any local embedding — no hosted model in the required path), then query.
3. **Reference-based scoring, computed in the example.** recall@k, precision@k, MRR against the answer key, printed per query and aggregated. Plain arithmetic over ids, so a reader can see exactly what each number counts.
4. **The reproducibility assertion, as a test.** Run the whole thing twice in one process and assert the score report is identical — the example proves its own claim instead of stating it. Then re-import and run again, so the claim covers "same corpus, fresh database", not just "same server".
5. **ACL as part of the score, not a footnote.** The same query asked by two identities has two different gold sets, because they can read different documents. A retriever that returns a document its caller may not read fails the eval outright rather than scoring slightly worse — that is the measurement no public RAG eval currently offers, and it is one `/_mock/users` token away here.
6. **A CI job.** `pytest -rs` already runs the suite on every push; the eval joins it as a gate with no secrets, no network beyond loopback, and no flake surface from someone else's outage.

Optional, once the above runs: a thin adapter that feeds the same answer key to whichever framework a team has already standardized on (promptfoo, DeepEval, RAGAS). Optional because the point is that this needs no framework — and because the required path must stay runnable with no Node and no key.

## What the determinism actually rests on

Worth stating in the README precisely, because "deterministic" is a claim readers should be able to check:

- ids and timestamps are derived from stable keys — a document's own key, a container's name, an author's address — not generated per run, so a gold id keeps pointing at the same document across runs, across pages of the same listing, and across a re-import of the same corpus
- responses are synthesized from the record you loaded, so nothing enters a response that the corpus did not put there
- pagination is stable, so a crawl that stops early and one that runs to exhaustion agree on what they saw
- the ACL is a property of the corpus, so per-identity gold sets are computable rather than guessed

## The caveat to bake in

Retrieval determinism is airtight. **LLM-judge determinism is not** — same prompt, same temperature, different day, different score. Keep the graded path non-judge. If the example shows a generated answer at all, it shows it as illustration, unscored, and says so.

## Done when

- [ ] `examples/deterministic-rag-eval/` runs end to end with no credentials and no network beyond the local server
- [ ] two consecutive runs, and a run after a re-import, produce identical score reports — asserted, not claimed
- [ ] per-identity gold sets are scored, and a document served to the wrong caller fails the run
- [ ] the README states what determinism rests on and where it stops
- [ ] a CI job runs it on every push

## Touchpoints

- [ ] `examples/deterministic-rag-eval/` — corpus, answer key, runnable script, README
- [ ] `pyproject.toml` — an `eval` extra for the local retriever/scoring dependencies
- [ ] `.github/workflows/ci.yml` — the eval job
- [ ] `README.md` — the examples table gains the eval entry
- [ ] `CONTRIBUTING.md` — the extras table (:68) gains `.[eval]` and what it unlocks

Start with the named touchpoints: examples/deterministic-rag-eval/, pyproject.toml, .github/workflows/ci.yml, README.md, and CONTRIBUTING.md. Run the existing pytest -rs suite first, then review how the example can use a local retriever and backlot.mock_server(records=...). Done means repeat and re-import runs have identical scores, ACL failures are caught, documentation is updated, and CI runs it without credentials or external network access.

Written by the indexing model from the issue text.