huggingface/tokenizers
View on GitHub💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
- Stars
- 11k
- Forks
- 1.2k
- Open beginner issues
- 0
- Indexed issues
- 120
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
- Dominant language
- Rust
- License
- Apache-2.0
- Last GitHub push
- Sep 15, 2026
- Latest indexed
- Sep 17, 2026
- Contributing guide
- Contributing guide
- Code of conduct
- No code of conduct
- Beginner labels
- No beginner labels indexed
-
huggingface/tokenizers#203 · 2 comments · 3 reactions · 0 assignees ·
-
Feature Request Stale
huggingface/tokenizers#659 · 3 comments · 0 reactions · 0 assignees ·
-
bytefallback Feature Request
huggingface/tokenizers#1407 · 5 comments · 1 reaction · 0 assignees ·
-
Feature Request planned Stale
huggingface/tokenizers#1473 · 9 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1495 · 12 comments · 2 reactions · 0 assignees ·
-
Feature Request training
huggingface/tokenizers#1515 · 4 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1519 · 8 comments · 2 reactions · 0 assignees ·
-
How to allow the merging of consecutive newline tokens \n when training a byte-level bpe tokenizer? Openbug
huggingface/tokenizers#1534 · 11 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1546 · 10 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1556 · 5 comments · 0 reactions · 0 assignees ·
-
Stale
huggingface/tokenizers#1559 · 5 comments · 0 reactions · 0 assignees ·
-
Decode regression Opendecoding performance
huggingface/tokenizers#1564 · 13 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1566 · 2 comments · 0 reactions · 0 assignees ·
-
Feature Request
huggingface/tokenizers#1573 · 4 comments · 1 reaction · 0 assignees ·
-
huggingface/tokenizers#1575 · 1 comment · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1576 · 1 comment · 0 reactions · 0 assignees ·
-
Stale
huggingface/tokenizers#1579 · 1 comment · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1581 · 6 comments · 1 reaction · 0 assignees ·
-
huggingface/tokenizers#1596 · 2 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1601 · 4 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1611 · 0 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1612 · 3 comments · 0 reactions · 0 assignees ·
-
.NET bindings Open
huggingface/tokenizers#1615 · 0 comments · 3 reactions · 0 assignees ·
-
huggingface/tokenizers#1619 · 6 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1624 · 1 comment · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1625 · 1 comment · 0 reactions · 0 assignees ·
-
Feature Request
huggingface/tokenizers#1627 · 5 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1628 · 2 comments · 2 reactions · 0 assignees ·
-
huggingface/tokenizers#1634 · 6 comments · 0 reactions · 0 assignees ·
-
bug
huggingface/tokenizers#1636 · 3 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1637 · 1 comment · 0 reactions · 0 assignees ·
-
Feature Request
huggingface/tokenizers#1656 · 1 comment · 1 reaction · 0 assignees ·
-
bug
huggingface/tokenizers#1663 · 3 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1654 · 0 comments · 0 reactions · 0 assignees ·
-
Feature Request
huggingface/tokenizers#1655 · 2 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1658 · 2 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1682 · 2 comments · 1 reaction · 0 assignees ·
-
huggingface/tokenizers#1683 · 4 comments · 10 reactions · 0 assignees ·
-
huggingface/tokenizers#1694 · 1 comment · 7 reactions · 0 assignees ·
-
huggingface/tokenizers#1695 · 0 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1696 · 0 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1698 · 4 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1700 · 1 comment · 0 reactions · 0 assignees ·
-
Thread safe? Open
huggingface/tokenizers#1726 · 6 comments · 0 reactions · 0 assignees ·
-
Slow compile times Open
huggingface/tokenizers#1730 · 0 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1738 · 2 comments · 0 reactions · 0 assignees ·
-
huggingface/tokenizers#1741 · 3 comments · 0 reactions · 0 assignees ·
-
Feature Request
huggingface/tokenizers#1751 · 4 comments · 27 reactions · 0 assignees ·
-
huggingface/tokenizers#1759 · 5 comments · 4 reactions · 0 assignees ·
-
Feature Request
huggingface/tokenizers#1803 · 4 comments · 4 reactions · 0 assignees ·