microsoft / microsoft/onnxruntime-genai

Unsupported tokenizer class for microsoft/OptiMind-SFT

Open
#1,987 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

I bumped the same issue ``Unsupported tokenizer class: TokenizersBackend``.

I created the following model ``python -m onnxruntime_genai.models.builder -m microsoft/OptiMind-SFT -o microsoft/OptiMind -p int4 -e cuda -c cache`` and tries it with following code:

```python
import onnxruntime_genai as og

model = og.Model(".")
tokenizer = og.Tokenizer(model)
stream = tokenizer.create_stream()

# Set the max length to something sensible by default,
# since otherwise it will be set to the entire context length
search_options = {}
search_options['max_length'] = 2048
search_options['batch_size'] = 1

chat_template = '<|user|>\n{input} <|end|>\n<|assistant|>'

text = "dummy example for somebody willing to learn python"
prompt = f'{chat_template.format(input=text)}'

input_tokens = tokenizer.encode(prompt)

params = og.GeneratorParams(model)
params.set_search_options(**search_options)
generator = og.Generator(model, params)

print("Output: ", end='', flush=True)

generator.append_tokens(input_tokens)
while not generator.is_done():
generator.generate_next_token()
new_token = generator.get_next_tokens()[0]
print(stream.decode(new_token), end='', flush=True)

print()
del generator
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the `python -m onnxruntime_genai.models.builder` command for `microsoft/OptiMind-SFT`, then run the shown `og.Tokenizer` and generation snippet. Trace the tokenizer entry point where `TokenizersBackend` is rejected; the issue is done when the built model loads and the example can generate output without the unsupported-tokenizer error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.