[Bug] tiktoken 0.9.0 ignores TIKTOKEN_OFFLINE and TIKTOKEN_CACHE_DIR, still attempts network download
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
Bug Description
在离线环境(无互联网连接)下,tiktoken==0.9.0 即使已经设置了 TIKTOKEN_CACHE_DIR 环境变量且本地缓存文件完整,仍然会尝试从网络下载编码文件,导致 requests.exceptions.ConnectionError 异常。
降级到 tiktoken==0.5.0 后问题消失,说明 0.9.0 版本的缓存机制存在回归。
Steps to Reproduce
-
准备环境(在有网机器上):
# 下载编码文件到默认缓存目录 mkdir -p ~/.cache/tiktoken cd ~/.cache/tiktoken wget https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken wget https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken wget https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken wget https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken -
切换到离线机器(断开网络)
-
设置环境变量
export TIKTOKEN_OFFLINE=1
export TIKTOKEN_CACHE_DIR=~/.cache/tiktoken
4. **运行测试代码:**
```bash
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(enc.encode("Hello"))
Expected Behavior
应直接使用本地缓存文件,不发送任何网络请求,正常输出 [9906, 37556]。
Actual Behavior
抛出 requests.exceptions.ConnectionError,尝试访问网络:
HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443):
Max retries exceeded with url: /encodings/cl100k_base.tiktoken
Workaround
降级到 tiktoken==0.5.0 可正常离线使用:
pip install tiktoken==0.5.0
Environment
-
tiktoken version: 0.9.0
-
Python version: 3.12.13 (Anaconda)
-
OS: Windows 10 / Ubuntu 22.04
-
Installation method: pip install tiktoken
Additional Context
在 0.5.0 版本中,离线缓存机制正常工作。升级到 0.9.0 后出现问题,即使在环境变量 TIKTOKEN_OFFLINE=1 和 TIKTOKEN_CACHE_DIR 都已正确设置的情况下,仍然会发起网络请求。
从调用栈可以看出,tiktoken.load.read_file() 函数仍然尝试通过 requests.get() 下载文件,而非优先使用本地缓存。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tiktoken.load.read_file(), where the issue reports that requests.get() is still used despite TIKTOKEN_OFFLINE and TIKTOKEN_CACHE_DIR. Reproduce the supplied offline example with a cached cl100k_base.tiktoken file; done means the local cache is used without a network request and encoding “Hello” produces [9906, 37556].
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100