EleutherAI / EleutherAI/pythia
Weird inconsistency in Tokenizer vocabulary
- Dominant language
- Jupyter Notebook
- Stars
- 2.9k
- Forks
- 224
- PR merge metrics
- No merged PRs in 30d
Description
Hello everyone!
I found a weird inconsistency in the tokenizer vocabulary. I wanted to ask why this could be happening.
I have loaded a tokenizer from HF:
```
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-160m")
```
If I run
```
tokenizer.encode("\u200b")
```
The output is `[12882]`. However, taking a look at the vocabulary used for training ([here](https://github.com/EleutherAI/pythia/blob/main/utils/20B_tokenizer.json)), I cannot find the token `\u200b` and the token id corresponds to a different string
```
"\u00e2\u0122\u012d": 12882,
```
This seems to generally happen with unicode characters.
Why could this be happening?? I just want to make sure that the tokenizer I use for training is equivalent to the HF tokenizers since my training (as anticipated in your README) results in a weird tokenizer.
Thanks a lot :)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.