Look into n-gram entropy

From Bill:

> `(word1, frequncy), (word2, frequency), ...`
> then trying to measure how far that distribution is from uniform
> one simple nice way is entropy
> `P(word1)*log(P(word1)) + P(word2)*log(P(word2)) + ...`
> where P(word1) is just frequency of word1 / total words
>
> It's nice because it measures how "unpredictable" the signal is. If most words are zero, and only a few words are common, then it's predictable. Or, if all words are exactly the same, then it's predictable. But if it's crazy town, then it's not predictable.

This seems like a decent resource: http://normal-extensions.com/2013/08/04/entropy-for-n-grams/

El issue no menciona archivos, pruebas ni puntos de entrada. Empieza leyendo el recurso enlazado entropy-for-n-grams y aclara cómo deben convertirse las frecuencias de palabras en probabilidades y entropía. El issue no especifica qué significa Done; establece la salida prevista y la validación antes de la implementación.

Escrito por el modelo de indexación a partir del texto del issue.