huggingface / huggingface/course

When training new tokenizer, why vocab size setting doesn't work?

Open
#801 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
4.2k
Forks
1.4k
Avg merge
13m
Merged PRs (30d)
1

Description

follow https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt

I am tring to replicate result using train_new_from_iterator

old_tokenizer = AutoTokenizer.from_pretrained('gpt2')

corpus = [
"This is the Hugging Face Course.",
"This chapter is about tokenization.",
"This section shows several tokenizer algorithms.",
"Hopefully, you will be able to understand how they are trained and generate tokens.",
]

tokenizer = old_tokenizer.train_new_from_iterator(corpus, vocab_size=50)

But the reuslt is totally different, and the tokenizer.vocab_size is 257 which is not 50. Why?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.