huggingface / huggingface/tokenizers
Why the tokenizer is slower than tiktoken?
Open
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Hi, I tried to use the GPT2 tokenizer of HF and TikToken, but I found TikToken is faster than HF. Could you explain why this might happen?
Contributor guide
Assessment
This issue has not been assessed yet.