nadavWeisler / nadavWeisler/MedCodeTranslator
[CI] Make search benchmarks a required quality gate
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 7m
- Merged PRs (30d)
- 7
Description
Problem summary
Search quality benchmarks exist (agents/search-benchmarker/, data/benchmarks/) but CI runs them with continue-on-error: true. The README precision@1 / precision@5 table is empty (—).
Affected area
.github/workflows/ci.yml.github/workflows/search-quality.ymlagents/search-benchmarker/run_benchmark.jsREADME.md
Expected behavior
Benchmark regressions fail CI. README displays current benchmark scores from the latest successful run.
Actual behavior
Benchmark failures do not block merges. Public docs do not show measured search quality.
Acceptance criteria
- Benchmark job fails CI on precision@1 or precision@5 regression below defined thresholds.
- Benchmark artifacts are published or summarized in CI output.
- README table is populated from benchmark results (manual or automated).
- Per-scheme thresholds are documented in
agents/search-benchmarker/README.md.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .github/workflows/ci.yml and .github/workflows/search-quality.yml, then inspect agents/search-benchmarker/run_benchmark.js and its README. Run the existing benchmark to trace precision@1 and precision@5 results and artifact handling. Done means regressions below documented per-scheme thresholds fail CI, results are published or summarized, and README.md shows current scores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, javascript, typescript
- Domain
- ci-cd, documentation, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100