huggingface / huggingface/course

Remove information about WordPiece lacking official implementation

Open
#539 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
4.2k
Forks
1.4k
Avg merge
13m
Merged PRs (30d)
1

Description

The course Website stated that WordPiece tokenizer's implemnetation has never been open-sourced:
https://huggingface.co/course/chapter6/6?fw=pt
It has been:
https://ai.googleblog.com/2021/12/a-fast-wordpiece-tokenization-system.html
and 🤗 Transformers already use it:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/tokenization_bert_tf.py
https://github.com/tensorflow/text/blob/master/tensorflow_text/python/ops/bert_tokenizer.py

Therefore I think the information should be rephrased.

I can do this task if it's possible.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.