huggingface / huggingface/course
Incorrect example explanation
- Dominant language
- MDX
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
The description of the unigram tokenization unigram in the article seems to be incorrect? see [this](https://huggingface.co/learn/llm-course/chapter6/7#tokenization-algorithm)
> Here are the frequencies of all the possible subwords in the vocabulary:
> `("h", 15) ("u", 36) ("g", 20) ("hu", 15) ("ug", 20) ("p", 17) ("pu", 17) ("n", 16)
("un", 16) ("b", 4) ("bu", 4) ("s", 5) ("hug", 15) ("gs", 5) ("ugs", 5)`
> The tokenization probability of ["p", "u", "g"] for "pug" is 5/210 * 36/210 * 20/210
Shouldn't it be 37/210 * 36/210 * 20/210? I'm also a beginner so I'm not sure if he is correct...
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.