huggingface/tokenizers.js

🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers

Stars: 57
Forks: 11
Open beginner issues: 0
Indexed issues: 2

Dominant language: TypeScript
License: Apache-2.0
Last GitHub push: Sep 7, 2026
Latest indexed: Sep 17, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Open All

2 open issues indexed

Add offset mapping support for character-level token alignment Open

huggingface/tokenizers.js#16 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
tokenizers.js ignores byte_fallback for Unigram models Open

enhancement

huggingface/tokenizers.js#26 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026

A short digest of beginner-friendly GitHub issues.