aRustyDev / aRustyDev/ccgram

Run lookup vs neural vs hybrid comparison

Open
#14 0 comments 0 reactions 0 assignees View on GitHub
evaluation experiment
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Run a comprehensive comparison of all three generation approaches on the test set.

## Success Criteria
- [ ] All three approaches evaluated on same test set
- [ ] Metrics computed for each: accuracy, BLEU, latency
- [ ] Statistical significance tests where applicable
- [ ] Results visualized (tables, charts)
- [ ] Reproducible experiment script

## Approaches to Compare
1. **Pure Lookup**: N-gram only, no fallback
2. **Pure Neural**: Neural model only, no lookup
3. **Hybrid**: Lookup with neural fallback (varying thresholds)

## Metrics Matrix
| Approach | Exact Match | Type Acc | BLEU-4 | P50 Latency | P99 Latency |
|----------|-------------|----------|--------|-------------|-------------|
| Lookup | | | | | |
| Neural | | | | | |
| Hybrid | | | | | |

## Experiment Config
- Test set: data/splits/test.jsonl
- Thresholds to test: [0.5, 0.6, 0.7, 0.8, 0.9]
- Hardware: Document CPU/GPU used

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.