Security: Cache Poisoning via Shared Temp Directory with Optional Hash Verification

Summary

Two related vulnerabilities in tiktoken/load.py allow local cache poisoning of BPE tokenizer data on shared systems, potentially affecting tokenization integrity for all users on a multi-user machine.

Vulnerability 1: Cache Poisoning via Shared Temp Directory (CWE-377/CWE-379)

File: tiktoken/load.py (lines 45, 77)
Severity: MEDIUM (CVSS 5.5)

When no TIKTOKEN_CACHE_DIR environment variable is set, tiktoken caches downloaded BPE data in tempfile.gettempdir() + "/data-gym-cache" — typically /tmp/data-gym-cache/ on Linux systems.

The directory is created with os.makedirs(cache_dir, exist_ok=True) with no explicit permission restrictions. On multi-user Linux systems, /tmp is world-writable, meaning:

1. A local attacker can pre-create /tmp/data-gym-cache/ with permissive ACLs before any legitimate user runs tiktoken
2. Cache filenames are predictable: hashlib.sha1(url.encode()).hexdigest() (line 52)
3. An attacker can compute the expected filename for any encoding (e.g., cl100k_base) and place a poisoned BPE file there
4. Subsequent users loading the same encoding will read the attacker-controlled file

Proof of Concept

# As attacker on shared Linux system:
mkdir -p /tmp/data-gym-cache
python3 -c "
import hashlib
url = 'https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken'
print(hashlib.sha1(url.encode()).hexdigest())
" 
# Output: predictable hash filename
# Write malicious BPE data to /tmp/data-gym-cache/<hash>

# As victim on same system:
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')  # Loads poisoned cache
enc.encode('sensitive text')  # Produces attacker-controlled token IDs

Vulnerability 2: Optional Hash Verification Allows Silent Cache Tampering (CWE-345)

File: tiktoken/load.py (line 58)
Severity: MEDIUM (CVSS 5.3)

The cache loading logic contains:

if expected_hash is None or check_hash(data, expected_hash):

When expected_hash is None, cached files are loaded without any integrity verification. While OpenAI's built-in encodings supply hashes, third-party or custom encodings that call load_tiktoken_bpe() without providing a hash get zero protection against tampered cache files.

Combined with Vulnerability 1, this means poisoned cache files for custom encodings are silently accepted with no warning or error.

Impact

• Tokenization integrity: Poisoned BPE data produces incorrect token IDs, which could cause:
  • Degraded model performance or nonsensical outputs
  • Subtle data corruption in ML pipelines
  • Potential for adversarial token manipulation in security-sensitive applications
• Scope: Any multi-user Linux system (CI/CD servers, shared dev machines, containers with shared /tmp)
• For custom encodings: Complete bypass of integrity checks when hashes are not supplied

Suggested Fixes

1. Use a user-specific cache directory by default (e.g., ~/.cache/tiktoken via platformdirs or os.path.expanduser)
2. Set restrictive permissions on the cache directory (e.g., 0o700)
3. Require hash verification or emit a warning when loading cache files without hash validation
4. Use tempfile.mkdtemp() or similar to create unique per-user temp directories if /tmp must be used

Reporter

Conner Webber (conner.webber000@gmail.com)

Start in tiktoken/load.py around the cache-directory and cache-loading logic identified at lines 45-77. Review how the default temporary directory, predictable filenames, and optional expected_hash handling interact; done means addressing shared-directory poisoning and unverified custom-encoding cache loads while preserving valid cache behavior.