jonathanbratt / jonathanbratt/RBERT
Rewrite and Speed Up Tokenizer
Open
- Dominant language
- R
- Stars
- 166
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
As an RBERT user, I'd like the tokenizer to be as fast as it can be, so that I don't have to wait for this step more than is absolutely necessary.
First thing to check: Does `keras::text_tokenizer` (and friends) do what we need? If so, we should be able to `save_text_tokenizer()` when the model is downloaded for #51.
Contributor guide
Assessment
This issue has not been assessed yet.