EleutherAI / EleutherAI/pythia

Weird inconsistency in Tokenizer vocabulary

Open
#151 1 comment 0 reactions 1 assignee Claimed by @haileyschoelkopf View on GitHub
Dominant language
Jupyter Notebook
Stars
2.9k
Forks
224
PR merge metrics
No merged PRs in 30d

Description

Hello everyone!

I found a weird inconsistency in the tokenizer vocabulary. I wanted to ask why this could be happening.

I have loaded a tokenizer from HF:

```
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-160m")
```

If I run

```
tokenizer.encode("\u200b")
```

The output is `[12882]`. However, taking a look at the vocabulary used for training ([here](https://github.com/EleutherAI/pythia/blob/main/utils/20B_tokenizer.json)), I cannot find the token `\u200b` and the token id corresponds to a different string

```
"\u00e2\u0122\u012d": 12882,
```

This seems to generally happen with unicode characters.

Why could this be happening?? I just want to make sure that the tokenizer I use for training is equivalent to the HF tokenizers since my training (as anticipated in your README) results in a weird tokenizer.

Thanks a lot :)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.