jonathanbratt / jonathanbratt/RBERT

Save tokenizer as part of model

Open
#51 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
R
Stars
166
Forks
19
PR merge metrics
No merged PRs in 30d

Description

The tokenizer for a given model is deterministic (it only depends on the vocab file + whether it's cased). Producing the tokenizer takes 100x as long as loading a pre-processed tokenizer (about 4 s vs 40 ms for bert_base_uncased).

Save the tokenizer as part of the download process. If a model has a vocab but not a tokenizer, save a tokenizer once and then use it going forward (for backward compatibility with things that are already downloaded).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.