tryAGI/Tiktoken
View on GitHubHigh-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
- Stars
- 90
- Forks
- 7
- Open beginner issues
- 0
- Indexed issues
- 2
- Avg merge
- 17h 57m
- Merged PRs (30d)
- 3
- Dominant language
- C#
- License
- MIT
- Last GitHub push
- Sep 11, 2026
- Latest indexed
- Sep 19, 2026
- Contributing guide
- Contributing guide
- Code of conduct
- Code of conduct
- Beginner labels
- No beginner labels indexed
2 open issues indexed
Loading issues
-
EncodeNative reads post-special-token segments from offset 0 on net8+ (silent token corruption) Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100