CosmicHorrorDev / CosmicHorrorDev/rust_text_classifier

Try out different tokenizers?

Open
#6 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13
Forks
1
PR merge metrics
No merged PRs in 30d

Description

The current tokenizer is pretty unaware of the structure of the text. Situations to improve upon would be

## tokenizing links

Something like `http://www.google.com/useless/junk` gets transformed `[http, www, google, com, useless, junk]` when I think it would be better to just be `[http://www.google.com]` since talking about just http and https is common in rust and some links like `docs` or `github.com` are very good indicators

It also allows for actually using links like `v.reddit.com` and `i.reddit.com` that lose a lot of structure when tokenized

## tokenizing code

Code blocks are very common in r/rust posts yet much of the syntax is ignored when tokenizing. It would be good to either recognize the code block and retain certain information normally stripped out (like `::` or `->` for example), or it would likely just be enough to recognize a code block and classify if it's rust (since that would be very unlikely to see for the rust game

## tokenizing reddit specific things

This is primarily for certain reddit specific things like `/u/username` or `/r/subreddit`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.