Brohammad / Brohammad/VoxForge
Implement scripts/knowledge_gate.py and fail CI on KB benchmark regressions
- Dominant language
- Python
- Stars
- 4
- Forks
- 1
- Avg merge
- 5d 13m
- Merged PRs (30d)
- 10
Description
## Summary
Onboarding already has `scripts/eval_gate.py` in CI. Knowledge-base benchmarks are documented with thresholds but the gate script is still marked future.
## Context
- `docs/benchmarks/knowledge-base.md` (citation p95 < 5 ms, search p95 < 100 ms, zero org-isolation failures)
- `scripts/eval_gate.py` + existing `eval-gate` CI job (pattern to copy)
- `scripts/benchmark_knowledge_base.py`
## Acceptance criteria
- [ ] `python scripts/knowledge_gate.py` exits 0 on current mock benchmarks
- [ ] Parses JSON from the knowledge-base benchmark
- [ ] CI job fails when thresholds regress (not `continue-on-error` forever)
- [ ] Unit test for threshold parsing
- [ ] Docs updated so “future” is removed
Distinct from #25 (nightly live-provider smoke).
Contributor guide
Research direction
Start with scripts/eval_gate.py and the existing eval-gate CI job to understand the project’s gate pattern, then inspect scripts/benchmark_knowledge_base.py and docs/benchmarks/knowledge-base.md for the JSON shape and thresholds. Add the unit test for threshold parsing and update the documentation; done means the knowledge gate passes current mock benchmarks and the CI job fails on threshold regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, documentation, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100