epfl-dlab / epfl-dlab/transformers-CFG
Adding support for XLMRobertaTokenizerFast
Open
- Dominant language
- Python
- Stars
- 139
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Description
Unfortunately, transformers-CFG fails for Huggingface models that use a XLMRobertaTokenizerFast tokenizers:
> Traceback (most recent call last):
> [...]
> AssertionError: Tokenizer not supported: XLMRobertaTokenizerFast, supported tokenizers: {, , , , , , , }
What does it take to add support for XLMRobertaTokenizerFast? If it make sense conceptually, I am interested in contributing to the addition of this tokenizer.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.