[FEA] Byte Pair Encoding Tokenizer
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
We should add byte pair encoding tokenizer to cuDF. Like our [subword-tokenizer](https://docs.rapids.ai/api/cudf/21.12/api_docs/api/cudf.core.subword_tokenizer.SubwordTokenizer.__call__.html) adds a bridge to Bert link models. Byte Pair EncodingTokenizer is used by `roberta`, `gpt-2` , `gpt-3` and will give us a bridge to a lot of DL models.
We should focus porting a pre-trained tokenizer first.
**Describe the solution you'd like**
The implimentation should follow [GPT-2 tokenizer ](https://huggingface.co/transformers/_modules/transformers/models/gpt2/tokenization_gpt2.html#GPT2Tokenizer) but should be extendable to the robert-a , `gpt-3`, `megatron` etc. We should follow the HuggingFace API for this.
**Algorithim:**
1. Add an identifier `()` at the end of each word to identify the end of a word and then calculate the word frequency in the text.
2. Split the word into characters and then calculate the character frequency.
3. From the character tokens, for a predefined number of iterations, count the frequency of the consecutive byte pairs and merge the most frequently occurring byte pairing.
4. Keep iterating until you have reached the iteration limit (set by you) or until you have reached the token limit.
Ref: [Link](https://www.freecodecamp.org/news/evolution-of-tokenization/)
**Additional context**
Best Explanation of Algorithm: https://leimao.github.io/blog/Byte-Pair-Encoding/
CC: @randerzander , @beckernick
Contributor guide
Assessment
This issue has not been assessed yet.