OpenNMT / OpenNMT/CTranslate2

Exception when using some T5 model

Open
#1,610 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
4.7k
Forks
537
Avg merge
12h 12m
Merged PRs (30d)
4

Description

When I was trying out some other T5 models and those models used the T5Tokenizer
for eg. ct2-transformers-converter --model Rostlab/prot_t5_xl_uniref50 --output_dir ./prot-t5-ct2/

There is an Exception: You're trying to run a Unigram model but you're file was trained with a different algorithm.

I noticed because in the TransformersConverter class, tokenizer_class = transformers. AutoTokenizer,
However, some models must use the T5Tokenizer

My solution:
change this code :

tokenizer_class = transformers.AutoTokenizer

to

if self._model_name_or_path == 'Rostlab/prot_t5_xl_uniref50':
    tokenizer_class = transformers.T5Tokenizer
else:
    tokenizer_class = transformers.AutoTokenizer

python 3.9.6
windows10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in python/ctranslate2/converters/transformers.py at the TransformersConverter tokenizer selection described in the issue. Reproduce the conversion command for Rostlab/prot_t5_xl_uniref50 and inspect how the tokenizer is loaded. Done means the affected T5 model converts without the reported Unigram exception, while existing model conversion behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.