huggingface / huggingface/tokenizers

Disable pretty-print when saving tokenizer.json files

Open
#1,656 1 comment 1 reaction 0 assignees View on GitHub
Feature Request
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

### Feature request

As the vocabulary of newer models, like Llama 3 or Gemma, increases in size, so does the size of the tokenizer, which includes the vocabulary as JSON (and merges for BPE tokenizers). Pretty-printing these files for serialization introduces a significant overhead as whitespace around the vocabulary and/or merges is added to the file.

This issue is even worse after the new BPE serialization update, which replaces merges like `"s1 s2"` with `["s1", "s2"]`, which is now formatted to be on separate lines:

![image](https://github.com/user-attachments/assets/6f6ed067-5f93-4708-b56f-da44c0a06d8e)

From quick testing, not pretty-printing the tokenizer.json reduces the file size from 17MB to 7MB.

Understandably, pretty-printing the file can help with debugging, but for those cases, it's probably better that the default is not formatted (and have a flag for outputting with formatting).

cc @ArthurZucker
(PS: I can move this to `huggingface/tokenizers` if it is more applicable there.

### Motivation

To reduce the file sizes (and bandwidth) of downloading, serializing, and uploading these files. In particular, this will greatly benefit Transformers.js users, where bandwidth is important.

### Your contribution

-

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.