HuggingFace Tokenizers compatibility
Open
- Dominant language
- Python
- Stars
- 596
- Forks
- 60
- Avg merge
- 19h 23m
- Merged PRs (30d)
- 1
Description
Hi, I have been trying to get SeqIO to work with HuggingFace's tokenizers for a bit but have been running into trouble with non-t5 based tokenizers. Specifically, it seems that, because they are not sentencepiece tokenizers, tokenizers for models such as GPT-2 are incompatible with SeqIO's `SentencePieceVocabulary` as they only have the vocab files:
```
{
'vocab_file': 'vocab.json',
'merges_file': 'merges.txt',
'tokenizer_file': 'tokenizer.json'
}
```
Is there a currently supported way to use these tokenizers with SeqIO? Or would I need to make my own vocab class?
Contributor guide
Assessment
This issue has not been assessed yet.