Analyze lookup hit rate and errors
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Deep dive into when lookup succeeds vs fails, and what kinds of errors occur.
## Success Criteria
- [ ] Hit rate by commit type computed
- [ ] Hit rate by context length analyzed
- [ ] Error categorization completed
- [ ] Patterns identified for lookup failures
- [ ] Recommendations for improving lookup coverage
## Analysis Dimensions
1. **By commit type**: Does lookup work better for "feat" vs "fix"?
2. **By scope**: Common scopes vs rare scopes
3. **By subject length**: Short vs long descriptions
4. **By vocabulary**: Common words vs rare/technical terms
## Error Categories
- [ ] Unknown token (OOV)
- [ ] Low frequency context
- [ ] Ambiguous continuation (high entropy)
- [ ] Grammar violations
- [ ] Semantic errors (syntactically valid but wrong meaning)
## Deliverables
- Error analysis report
- Visualizations of hit rate distributions
- Recommendations for dataset or model improvements
Contributor guide
Assessment
This issue has not been assessed yet.