jonathanbratt / jonathanbratt/RBERT
Save tokenizer as part of model
Open
- Dominant language
- R
- Stars
- 166
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
The tokenizer for a given model is deterministic (it only depends on the vocab file + whether it's cased). Producing the tokenizer takes 100x as long as loading a pre-processed tokenizer (about 4 s vs 40 ms for bert_base_uncased).
Save the tokenizer as part of the download process. If a model has a vocab but not a tokenizer, save a tokenizer once and then use it going forward (for backward compatibility with things that are already downloaded).
Contributor guide
Assessment
This issue has not been assessed yet.