Exception when using some T5 model
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 537
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
When I was trying out some other T5 models and those models used the T5Tokenizer
for eg. ct2-transformers-converter --model Rostlab/prot_t5_xl_uniref50 --output_dir ./prot-t5-ct2/
There is an Exception: You're trying to run a Unigram model but you're file was trained with a different algorithm.
I noticed because in the TransformersConverter class, tokenizer_class = transformers. AutoTokenizer,
However, some models must use the T5Tokenizer
My solution:
change this code :
tokenizer_class = transformers.AutoTokenizer
to
if self._model_name_or_path == 'Rostlab/prot_t5_xl_uniref50':
tokenizer_class = transformers.T5Tokenizer
else:
tokenizer_class = transformers.AutoTokenizer
python 3.9.6
windows10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in python/ctranslate2/converters/transformers.py at the TransformersConverter tokenizer selection described in the issue. Reproduce the conversion command for Rostlab/prot_t5_xl_uniref50 and inspect how the tokenizer is loaded. Done means the affected T5 model converts without the reported Unigram exception, while existing model conversion behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100