huggingface / huggingface/course
a question about the "6.tokenizers library"
Open
- Dominant language
- MDX
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
when i studied the "6. The tokenizers liabrary -- Unigram tokenization" , i couldn't understand the following
why the P("pu") = 5/210, shouldn't it be the 17 / 210, because the P("g") = 20 / 210 according to the frequency of g is 20.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.