CosmicHorrorDev / CosmicHorrorDev/rust_text_classifier
Try out different tokenizers?
- Dominant language
- Python
- Stars
- 13
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
The current tokenizer is pretty unaware of the structure of the text. Situations to improve upon would be
## tokenizing links
Something like `http://www.google.com/useless/junk` gets transformed `[http, www, google, com, useless, junk]` when I think it would be better to just be `[http://www.google.com]` since talking about just http and https is common in rust and some links like `docs` or `github.com` are very good indicators
It also allows for actually using links like `v.reddit.com` and `i.reddit.com` that lose a lot of structure when tokenized
## tokenizing code
Code blocks are very common in r/rust posts yet much of the syntax is ignored when tokenizing. It would be good to either recognize the code block and retain certain information normally stripped out (like `::` or `->` for example), or it would likely just be enough to recognize a code block and classify if it's rust (since that would be very unlikely to see for the rust game
## tokenizing reddit specific things
This is primarily for certain reddit specific things like `/u/username` or `/r/subreddit`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.