jonathanbratt / jonathanbratt/RBERT
Decide how to handle tokenization conventions
Open
- Dominant language
- R
- Stars
- 166
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
There are several tokenization conventions (e.g. the token used for padding, separating segments, etc.) that need to be specified when doing the wordpiece tokenization for BERT. Currently, some of these conventions are hard-coded in, while others are function parameters. We should decide on a consistent approach here.
Also, more clearly delineate what belongs in RBERT vs. wordpiece.
(https://github.com/jonathanbratt/wordpiece/issues/15)
Contributor guide
Assessment
This issue has not been assessed yet.