CI: wire the Librarian eval harness in as a merge gate (explicit 100% + optional recall@k)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 180
- Forks
- 137
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 21
Description
Follow-up from @northdpole's reviews on #925 and #937 (Module C - The Librarian).
The gap: the eval harness (scripts/evaluate_librarian.py) already fails loudly on an explicit-slice regression - it exits non-zero if the "explicit" slice drops below 100% on the golden dataset. The problem is nothing in CI actually runs it. make test only runs the unit tests, so today that 100% gate exists on paper but isn't enforced. A change could quietly break the explicit slice and still merge green.
What we'd like to do: wire the harness into CI so it guards merges automatically.
- Add a small subprocess test (or a
Makefiletarget) that runsevaluate_librarian.pyagainst the committed golden dataset (application/tests/librarian/fixtures/golden_dataset.json). If the explicit slice isn't at 100%, the check fails and blocks the merge. This runs fully offline - no API key needed. - Later, extend the same setup to cover the C.1 retriever's recall@k. That part needs live embeddings (a populated cache DB + an embedding API key), so it can either be an optional CI job or a documented manual pre-merge check.
The offline explicit gate is the priority since it's deterministic and needs no credentials.
Scope: Module C. Not blocking W1–W3 - this just makes sure the harness is doing its job before the heavier ML slices (W4+) land.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/evaluate_librarian.py and application/tests/librarian/fixtures/golden_dataset.json, then inspect how make test and the repository's CI currently run checks. Add an offline CI check that runs the harness against the committed golden dataset and fails when the explicit slice is below 100%. The optional recall@k check is later work and requires live embeddings, a populated cache DB, and an API key.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100