Sentencepiece base Language
Open
enhancement
feat / tokenizer
feat / transformer
new language
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
feature request:
Sentencepiece is the tokenizer used in XLNet.
I think if `Language` tokenize text with `sentencepiece`, the alignment process can be skipped and it make model efficient.
Contributor guide
Assessment
This issue has not been assessed yet.