Run lookup vs neural vs hybrid comparison
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Run a comprehensive comparison of all three generation approaches on the test set.
## Success Criteria
- [ ] All three approaches evaluated on same test set
- [ ] Metrics computed for each: accuracy, BLEU, latency
- [ ] Statistical significance tests where applicable
- [ ] Results visualized (tables, charts)
- [ ] Reproducible experiment script
## Approaches to Compare
1. **Pure Lookup**: N-gram only, no fallback
2. **Pure Neural**: Neural model only, no lookup
3. **Hybrid**: Lookup with neural fallback (varying thresholds)
## Metrics Matrix
| Approach | Exact Match | Type Acc | BLEU-4 | P50 Latency | P99 Latency |
|----------|-------------|----------|--------|-------------|-------------|
| Lookup | | | | | |
| Neural | | | | | |
| Hybrid | | | | | |
## Experiment Config
- Test set: data/splits/test.jsonl
- Thresholds to test: [0.5, 0.6, 0.7, 0.8, 0.9]
- Hardware: Document CPU/GPU used
Contributor guide
Assessment
This issue has not been assessed yet.