huggingface / huggingface/tokenizers
Special token gets tokenized while training tokenizer from scratch
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
@ArthurZucker I am trying to train a bytepiece tokenizer on my dataset. I have a list of words which I want to be treated as a single token. But when I train it and tokenize, I observe that the token gets split in tow parts. My end goal is to train a Roberta LM on my dataset.
`from tokenizers import BertWordPieceTokenizer, ByteLevelBPETokenizer
files = 'file.txt'
tokenizer = ByteLevelBPETokenizer( lowercase=True, )
tokenizer.train( files, vocab_size=100000, min_frequency=5, show_progress=True, special_tokens=["", "", "", "", "", "auto_part", "bokchoy"], )
tokenizer.save_model('bpe_piece') `
Test the tokenizer:
`from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained('bpe_piece')
print(tokenizer.tokenize('an bokchoy auto_part))`
Output should be ['an', 'bokchoy', 'auto_part']
But instead the output is ['an', 'Ġbok', 'choy', 'Ġauto', '_', 'part']
Contributor guide
Research direction
Start by running the provided ByteLevelBPETokenizer.train reproduction and then loading the saved model with RobertaTokenizer.from_pretrained. Trace how the listed special tokens are registered and applied during tokenization. Done means the documented words remain single tokens after training and loading, with coverage for the reported example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100