openai / openai/tiktoken

F-REQ: If the pip installer doesn't find Rust, it should install the pure python version of the tokenizer

Open
#227 2 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19.3k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Currently Tiktoken (and with it all the OpenAI related python libraries using it) cannot be installed on systems and platforms that cannot (or are forbidden to) install Rust. This is a big issue, and many times it was rised here.

See:
pyodide/pyodide#36
pyodide/pyodide#57
pyodide/pyodide#94
pyodide/pyodide#134
josephrocca/gpt-2-3-tokenizer#2
pyodide/pyodide#3875
pyodide/pyodide#3663
pyodide/pyodide-recipes#288
emscripten-forge/recipes#660
psymbio/tiktoken_rust_wasm
https://github.com/openai/tiktoken/issues/94#issuecomment-1773748693

There are already 2 pure python implementations of the tokenizer:

In the educational version:
https://github.com/openai/tiktoken/blob/main/tiktoken/_educational.py
In this fork, courtesy of @kechan:
https://github.com/kechan/tiktoken
As discussed here: https://github.com/openai/tiktoken/issues/36

Since everything is in place, the solution would be simple: If the pip installer doesn't find Rust, it should install the pure python version of the tokenizer.
Please consider it. Making Rust mandatory to use OpenAI api it's inconvenient and only making the API accessible to less users and companies. It is in the best interest of OpenAI make its tools as portable as possible, and Python it's the perfect language for this. Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading tiktoken/_educational.py and the existing pip installation and Rust-related packaging setup. Compare the educational implementation with the main tokenizer and determine how installation can select a pure-Python fallback when Rust is unavailable; done means affected systems can install and use tiktoken without Rust.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.