jonathanbratt / jonathanbratt/RBERT

Decide how to handle tokenization conventions

Open
#58 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
166
Forks
19
PR merge metrics
No merged PRs in 30d

Description

There are several tokenization conventions (e.g. the token used for padding, separating segments, etc.) that need to be specified when doing the wordpiece tokenization for BERT. Currently, some of these conventions are hard-coded in, while others are function parameters. We should decide on a consistent approach here.

Also, more clearly delineate what belongs in RBERT vs. wordpiece.
(https://github.com/jonathanbratt/wordpiece/issues/15)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.