huggingface / huggingface/tokenizers

Special token gets tokenized while training tokenizer from scratch

Open
#1,624 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

@ArthurZucker I am trying to train a bytepiece tokenizer on my dataset. I have a list of words which I want to be treated as a single token. But when I train it and tokenize, I observe that the token gets split in tow parts. My end goal is to train a Roberta LM on my dataset.
`from tokenizers import BertWordPieceTokenizer, ByteLevelBPETokenizer

files = 'file.txt'

tokenizer = ByteLevelBPETokenizer( lowercase=True, )
tokenizer.train( files, vocab_size=100000, min_frequency=5, show_progress=True, special_tokens=["", "", "", "", "", "auto_part", "bokchoy"], )

tokenizer.save_model('bpe_piece') `

Test the tokenizer:
`from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained('bpe_piece')
print(tokenizer.tokenize('an bokchoy auto_part))`
Output should be ['an', 'bokchoy', 'auto_part']
But instead the output is ['an', 'Ġbok', 'choy', 'Ġauto', '_', 'part']

Contributor guide

Open the contributing guide

Research direction

Start by running the provided ByteLevelBPETokenizer.train reproduction and then loading the saved model with RobertaTokenizer.from_pretrained. Trace how the listed special tokens are registered and applied during tokenization. Done means the documented words remain single tokens after training and loading, with coverage for the reported example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.