Build 1-gram / 2-gram / 3-gram tables
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Build n-gram frequency tables from the training data for lookup-based generation.
## Success Criteria
- [ ] 1-gram table: token → count
- [ ] 2-gram table: (token₁) → [(token₂, count), ...]
- [ ] 3-gram table: (token₁, token₂) → [(token₃, count), ...]
- [ ] Tables serialized efficiently (consider trie structure)
- [ ] Memory footprint documented
- [ ] Lookup time benchmarked (<1μs per lookup target)
## Implementation Notes
- Consider smoothing for unseen n-grams
- Store both raw counts and probabilities
- Prune low-frequency entries to control size
## Table Format
\`\`\`python
{
"unigram": {"feat": 15000, "fix": 12000, ...},
"bigram": {"feat": {"add": 5000, "implement": 3000, ...}, ...},
"trigram": {("feat", "add"): {"new": 2000, "support": 1500, ...}, ...}
}
\`\`\`
Contributor guide
Assessment
This issue has not been assessed yet.