openai / openai/tiktoken

[Bug] tiktoken 0.9.0 ignores TIKTOKEN_OFFLINE and TIKTOKEN_CACHE_DIR, still attempts network download

Open
#513 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19.3k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Bug Description

在离线环境(无互联网连接)下,tiktoken==0.9.0 即使已经设置了 TIKTOKEN_CACHE_DIR 环境变量且本地缓存文件完整,仍然会尝试从网络下载编码文件,导致 requests.exceptions.ConnectionError 异常。

降级到 tiktoken==0.5.0 后问题消失,说明 0.9.0 版本的缓存机制存在回归。

Steps to Reproduce
  1. 准备环境(在有网机器上):

    # 下载编码文件到默认缓存目录
    mkdir -p ~/.cache/tiktoken
    cd ~/.cache/tiktoken
    wget https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken
    wget https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken
    wget https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken
    wget https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken
    
  2. 切换到离线机器(断开网络)

  3. 设置环境变量

export TIKTOKEN_OFFLINE=1
export TIKTOKEN_CACHE_DIR=~/.cache/tiktoken
4. **运行测试代码:**
```bash
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(enc.encode("Hello"))
Expected Behavior

应直接使用本地缓存文件,不发送任何网络请求,正常输出 [9906, 37556]。

Actual Behavior

抛出 requests.exceptions.ConnectionError,尝试访问网络:

HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): 
Max retries exceeded with url: /encodings/cl100k_base.tiktoken
Workaround

降级到 tiktoken==0.5.0 可正常离线使用:

pip install tiktoken==0.5.0
Environment
  • tiktoken version: 0.9.0

  • Python version: 3.12.13 (Anaconda)

  • OS: Windows 10 / Ubuntu 22.04

  • Installation method: pip install tiktoken

Additional Context

在 0.5.0 版本中,离线缓存机制正常工作。升级到 0.9.0 后出现问题,即使在环境变量 TIKTOKEN_OFFLINE=1 和 TIKTOKEN_CACHE_DIR 都已正确设置的情况下,仍然会发起网络请求。

从调用栈可以看出,tiktoken.load.read_file() 函数仍然尝试通过 requests.get() 下载文件,而非优先使用本地缓存。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tiktoken.load.read_file(), where the issue reports that requests.get() is still used despite TIKTOKEN_OFFLINE and TIKTOKEN_CACHE_DIR. Reproduce the supplied offline example with a cached cl100k_base.tiktoken file; done means the local cache is used without a network request and encoding “Hello” produces [9906, 37556].

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.