epfl-dlab / epfl-dlab/transformers-CFG

Tokenizers not model-agnostic

Open
#122 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
139
Forks
23
PR merge metrics
No merged PRs in 30d

Description

I believe that the library's support models are based on tokenization and not model brand.

For example, using any of the latest Falcon models should work because they run on `PreTrainedTokenizerFast`, however, the following error arises:

```
NotImplementedError: Tokenizer not supported: PreTrainedTokenizerFast
```

This seems to be because the library recognizes models based on their path or title as opposed to their tokenizer type, and so `tiiuae/Falcon3-1B-Base` is unsupported, even though it should not be. The solution is to compare the loaded tokenizer to a list of supported tokenizer classes instead of focusing on naming conventions.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.