aRustyDev / aRustyDev/ccgram

Evaluate pure lookup generator

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
evaluation experiment good-first-experiment
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Evaluate the pure n-gram lookup generator on the test set to establish a baseline.

## Success Criteria
- [ ] Exact match accuracy measured
- [ ] Type/scope/subject accuracy measured separately
- [ ] Coverage: % of test cases where lookup has high confidence
- [ ] Error analysis: categorize failure modes
- [ ] Latency benchmarked (p50, p95, p99)

## Metrics
| Metric | Description |
|--------|-------------|
| Exact Match | Full commit message matches |
| Type Accuracy | Correct commit type |
| Scope Accuracy | Correct scope (when applicable) |
| BLEU-4 | N-gram overlap score |
| Coverage | % of cases with confidence > threshold |

## Deliverables
- Evaluation script in scripts/
- Results saved to experiments/lookup-baseline/
- Analysis notebook or markdown report

## This is a good first experiment!
Clear scope, measurable outcomes, introduces experiment infrastructure.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.