huggingface / huggingface/tokenizers
ByteLevelBPETokenizer output seems weird
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
I use the `ByteLevelBPETokenizer` to train a custom tokenizer for Amharic language (less-resource language).
```
tokenizer = ByteLevelBPETokenizer(lowercase=False)
tokenizer.train(files=paths, vocab_size=32000, min_frequency=3, special_tokens=[
"",
"",
"",
"",
"",
])
```
The merge.txt and vocab.json files I obtained are now not human readable.
```
áĪ ħ
áĪ °
ĠáĬ¥ áĬķ
ĠáĬ ¨
Ġáĭ Ń
áĬ Ń
ĠáĪ Ī
áį į
```
Also the encoding results in the same unreadable output
```
output = tokenizer.encode("አበበ በሶ በላ። ጫላ ጩቤ ጨበጠ፡፡")
print(output.ids, output.tokens, output.offsets)
>>>[0, 319, 5739, 2883, 4037, 303, 1631, 299, 5173, 506, 748, 11918, 363, 2] ['', 'áĬł', 'áīłáīł', 'Ġáīłáζ', 'ĠáīłáĪĭ', 'áį¢', 'ĠáĮ«', 'áĪĭ', 'ĠáĮ©', 'áī¤', 'ĠáĮ¨', 'áīłáĮł', 'áį¡áį¡', ''] [(0, 0), (0, 3), (3, 9), (9, 16), (16, 23), (23, 26), (26, 30), (30, 33), (33, 37), (37, 40), (40, 44), (44, 50), (50, 56), (0, 0)]
```
Is this the expected behavior? I will later use this to train a RoberTa model using the `run_language_modeling.py ` script.
Thanks
Contributor guide
Assessment
This issue has not been assessed yet.