huggingface / huggingface/course
Remove information about WordPiece lacking official implementation
Open
- Dominant language
- MDX
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
The course Website stated that WordPiece tokenizer's implemnetation has never been open-sourced:
https://huggingface.co/course/chapter6/6?fw=pt
It has been:
https://ai.googleblog.com/2021/12/a-fast-wordpiece-tokenization-system.html
and 🤗 Transformers already use it:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert_tf.py
https://github.com/tensorflow/text/blob/master/tensorflow_text/python/ops/bert_tokenizer.py
Therefore I think the information should be rephrased.
I can do this task if it's possible.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.