aRustyDev / aRustyDev/ccgram

Build 1-gram / 2-gram / 3-gram tables

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
data model
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Build n-gram frequency tables from the training data for lookup-based generation.

## Success Criteria
- [ ] 1-gram table: token → count
- [ ] 2-gram table: (token₁) → [(token₂, count), ...]
- [ ] 3-gram table: (token₁, token₂) → [(token₃, count), ...]
- [ ] Tables serialized efficiently (consider trie structure)
- [ ] Memory footprint documented
- [ ] Lookup time benchmarked (<1μs per lookup target)

## Implementation Notes
- Consider smoothing for unseen n-grams
- Store both raw counts and probabilities
- Prune low-frequency entries to control size

## Table Format
\`\`\`python
{
"unigram": {"feat": 15000, "fix": 12000, ...},
"bigram": {"feat": {"add": 5000, "implement": 3000, ...}, ...},
"trigram": {("feat", "add"): {"new": 2000, "support": 1500, ...}, ...}
}
\`\`\`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.