aRustyDev / aRustyDev/ccgram

Analyze lookup hit rate and errors

Open
#15 0 comments 0 reactions 0 assignees View on GitHub
ablation evaluation
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Deep dive into when lookup succeeds vs fails, and what kinds of errors occur.

## Success Criteria
- [ ] Hit rate by commit type computed
- [ ] Hit rate by context length analyzed
- [ ] Error categorization completed
- [ ] Patterns identified for lookup failures
- [ ] Recommendations for improving lookup coverage

## Analysis Dimensions
1. **By commit type**: Does lookup work better for "feat" vs "fix"?
2. **By scope**: Common scopes vs rare scopes
3. **By subject length**: Short vs long descriptions
4. **By vocabulary**: Common words vs rare/technical terms

## Error Categories
- [ ] Unknown token (OOV)
- [ ] Low frequency context
- [ ] Ambiguous continuation (high entropy)
- [ ] Grammar violations
- [ ] Semantic errors (syntactically valid but wrong meaning)

## Deliverables
- Error analysis report
- Visualizations of hit rate distributions
- Recommendations for dataset or model improvements

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.