Evaluate pure lookup generator
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Evaluate the pure n-gram lookup generator on the test set to establish a baseline.
## Success Criteria
- [ ] Exact match accuracy measured
- [ ] Type/scope/subject accuracy measured separately
- [ ] Coverage: % of test cases where lookup has high confidence
- [ ] Error analysis: categorize failure modes
- [ ] Latency benchmarked (p50, p95, p99)
## Metrics
| Metric | Description |
|--------|-------------|
| Exact Match | Full commit message matches |
| Type Accuracy | Correct commit type |
| Scope Accuracy | Correct scope (when applicable) |
| BLEU-4 | N-gram overlap score |
| Coverage | % of cases with confidence > threshold |
## Deliverables
- Evaluation script in scripts/
- Results saved to experiments/lookup-baseline/
- Analysis notebook or markdown report
## This is a good first experiment!
Clear scope, measurable outcomes, introduces experiment infrastructure.
Contributor guide
Assessment
This issue has not been assessed yet.