Unofficial bindings / ports in other languages

The following projects are not maintained by OpenAI. I cannot vouch that any of them are correct or safe to use. Use at your own risk.

Note that if a tokeniser fails to exactly match tiktoken's behaviour, you may get worse results when sampling from models, with no warning.

Javascript

• https://github.com/dqbd/tiktoken
• https://github.com/ceifa/tiktoken-node
• https://github.com/niieani/gpt-tokenizer
• The gpt-3-encoder package will work for most GPT-3 models. However, it will often appear to work for Codex or GPT-3.5 while actually being out of distribution, and will not at all work for GPT-4 or embeddings models.

Rust

• https://github.com/zurawiki/tiktoken-rs

Java

• https://github.com/eisber/tiktoken
• https://github.com/knuddelsgmbh/jtokkit

Ruby

• https://github.com/volition-co/tiktoken

C#

• https://github.com/dmitry-brazhenko/SharpToken

Go

• https://github.com/tiktoken-go/tokenizer
• https://github.com/pkoukk/tiktoken-go

PHP

• https://github.com/danny50610/bpe-tokeniser

Kotlin

• https://github.com/aallam/ktoken

Thanks to everyone for building useful things!

I'm happy to link to other projects in this comment.

Review the issue body and its listed external projects, checking that each link is relevant and that the tokeniser compatibility warnings remain accurate. Done means the unofficial bindings list is complete and clearly labeled as unmaintained by OpenAI; no repository file is named in the issue.