aRustyDev / aRustyDev/ccgram

Implement tiny neural fallback model

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
model
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Implement a small neural language model to handle cases where n-gram lookup has low confidence.

## Success Criteria
- [ ] Model architecture chosen (e.g., small transformer, LSTM)
- [ ] Model size < 50MB (target: runs on CPU efficiently)
- [ ] Inference time < 100ms on CPU
- [ ] Model can be conditioned on partial context
- [ ] Training script implemented
- [ ] Checkpointing and resumption supported

## Architecture Candidates
1. **Tiny Transformer**: 2-4 layers, 128-256 hidden, ~5M params
2. **LSTM**: 2 layers, 256 hidden, ~3M params
3. **Distilled from larger model**: Knowledge distillation approach

## Constraints
- Must run locally on CPU
- Must fit in memory alongside n-gram tables
- Latency acceptable for interactive use

## Interface
\`\`\`python
class NeuralFallback:
def generate(self, context: list[str], max_tokens: int = 50) -> str:
"""Generate continuation given context."""
...
\`\`\`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.