huggingface / huggingface/course
When training new tokenizer, why vocab size setting doesn't work?
- Dominant language
- MDX
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
follow https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt
I am tring to replicate result using train_new_from_iterator
old_tokenizer = AutoTokenizer.from_pretrained('gpt2')
corpus = [
"This is the Hugging Face Course.",
"This chapter is about tokenization.",
"This section shows several tokenizer algorithms.",
"Hopefully, you will be able to understand how they are trained and generate tokens.",
]
tokenizer = old_tokenizer.train_new_from_iterator(corpus, vocab_size=50)
But the reuslt is totally different, and the tokenizer.vocab_size is 257 which is not 50. Why?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.