coder/ai-tokenizer

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

Stars: 59
Forks: 6
Open beginner issues: 0
Indexed issues: 2

Dominant language: TypeScript
License: MIT
Last GitHub push: Dec 1, 2025
Latest indexed: Sep 15, 2026
Contributing guide: No contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Open All

2 issues indexed so far

Typescript error while counting tokens Open

coder/ai-tokenizer#1 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Quadratic time in bytePairMerge: one long run of repeated characters blocks for minutes Open

coder/ai-tokenizer#6 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026

A short digest of beginner-friendly GitHub issues.