google / google/seqio

HuggingFace Tokenizers compatibility

Open
#188 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
596
Forks
60
Avg merge
19h 23m
Merged PRs (30d)
1

Description

Hi, I have been trying to get SeqIO to work with HuggingFace's tokenizers for a bit but have been running into trouble with non-t5 based tokenizers. Specifically, it seems that, because they are not sentencepiece tokenizers, tokenizers for models such as GPT-2 are incompatible with SeqIO's `SentencePieceVocabulary` as they only have the vocab files:

```
{
'vocab_file': 'vocab.json',
'merges_file': 'merges.txt',
'tokenizer_file': 'tokenizer.json'
}
```
Is there a currently supported way to use these tokenizers with SeqIO? Or would I need to make my own vocab class?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.