Brohammad / Brohammad/VoxForge

Implement scripts/knowledge_gate.py and fail CI on KB benchmark regressions

Open
#32 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
4
Forks
1
Avg merge
5d 13m
Merged PRs (30d)
10

Description

## Summary

Onboarding already has `scripts/eval_gate.py` in CI. Knowledge-base benchmarks are documented with thresholds but the gate script is still marked future.

## Context

- `docs/benchmarks/knowledge-base.md` (citation p95 < 5 ms, search p95 < 100 ms, zero org-isolation failures)
- `scripts/eval_gate.py` + existing `eval-gate` CI job (pattern to copy)
- `scripts/benchmark_knowledge_base.py`

## Acceptance criteria

- [ ] `python scripts/knowledge_gate.py` exits 0 on current mock benchmarks
- [ ] Parses JSON from the knowledge-base benchmark
- [ ] CI job fails when thresholds regress (not `continue-on-error` forever)
- [ ] Unit test for threshold parsing
- [ ] Docs updated so “future” is removed

Distinct from #25 (nightly live-provider smoke).

Contributor guide

Open the contributing guide

Research direction

Start with scripts/eval_gate.py and the existing eval-gate CI job to understand the project’s gate pattern, then inspect scripts/benchmark_knowledge_base.py and docs/benchmarks/knowledge-base.md for the JSON shape and thresholds. Add the unit test for threshold parsing and update the documentation; done means the knowledge gate passes current mock benchmarks and the CI job fails on threshold regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, documentation, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.