Try out different tokenizers?

The current tokenizer is pretty unaware of the structure of the text. Situations to improve upon would be

## tokenizing links

Something like `http://www.google.com/useless/junk` gets transformed `[http, www, google, com, useless, junk]` when I think it would be better to just be `[http://www.google.com]` since talking about just http and https is common in rust and some links like `docs` or `github.com` are very good indicators

It also allows for actually using links like `v.reddit.com` and `i.reddit.com` that lose a lot of structure when tokenized

## tokenizing code

Code blocks are very common in r/rust posts yet much of the syntax is ignored when tokenizing. It would be good to either recognize the code block and retain certain information normally stripped out (like `::` or `->` for example), or it would likely just be enough to recognize a code block and classify if it's rust (since that would be very unlikely to see for the rust game

## tokenizing reddit specific things

This is primarily for certain reddit specific things like `/u/username` or `/r/subreddit`

El issue no nombra archivos ni tests. Empieza por localizar el tokenizer actual y sus tests, y después compara cómo se representan los enlaces, los bloques de código, la sintaxis de Rust y las rutas de Reddit. Se consideraría terminado cuando haya un alcance acordado y tests que demuestren que las estructuras seleccionadas conservan la información prevista.

Escrito por el modelo de indexación a partir del texto del issue.