Community Resource: AutoTikTokenizer - A Bridge Between TikToken and HuggingFace Tokenizers

Hi TikToken team! 👋

I wanted to share a community resource that might be helpful for TikToken users who also work with HuggingFace tokenizers. I've created AutoTikTokenizer, a lightweight library that allows loading any HuggingFace tokenizer as a TikToken-compatible encoder.

What it does:

• Enables using TikToken's fast tokenization with any HuggingFace tokenizer
• Preserves exact encoding/decoding compatibility with original tokenizers
• Simple drop-in usage similar to HuggingFace's AutoTokenizer

Quick example:

from autotiktokenizer import AutoTikTokenizer

# Load any HF tokenizer as a TikToken encoder
encoder = AutoTikTokenizer.from_pretrained('gpt2')
tokens = encoder.encode("Hello world!")
text = encoder.decode(tokens)

The library is available on PyPI (pip install autotiktokenizer) and is fully open source at: https://github.com/bhavnicksm/autotiktokenizer

I've tested it with several popular models including GPT-2, LLaMA, Mistral, and others. I hope this helps TikToken users who want to work with a broader range of tokenizers while keeping TikToken's performance benefits!

Feel free to check it out if you think it would be useful for the community. Happy to hear any feedback or suggestions!

[Note: This is purely a community contribution - I'm not affiliated with the TikToken team]

The issue body is a community announcement and names no tiktoken repository files, tests, or entry points; begin by reviewing the linked AutoTikTokenizer repository. There is no requested repository change or stated completion criterion, so confirm whether maintainers want this documented before proposing any implementation.