antirez / antirez/smaz

codebook with the most frequent ngrams in language/s

Ouverte
#11 0 commentaires 5 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
1.2k
Forks
133
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

I know this guy..;) (from Redis)
did you hand pick the codebook dictionary? how?
have you though about using the most frequent ngrams in language/s?
e.g the top (e.g 32) ngrams from [Norvig](http://norvig.com/mayzner.html)'s ngrams2,3,4,5,6,7,8,9.csv?
How do you optimally pick them for minimum overlap and better compression rates? i.e
`ation` and `tion` are the most common 4 and 5 letters long ngrams respectively, `tio` is the 6th most common 3 letters ngram.
I think you'd get much better/higher compression rates.

I wanna test it, but couldn't find any docs.
so what are these characters?
```
static char *Smaz_cb[241] = {
"\002s,\266", "\003had\232\002leW", "\003on \216", "", "\001yS",
"\002ma\255\002li\227", "\003or \260", "", "\002ll\230\003s t\277",
```

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.