microsoft / microsoft/onnxruntime-genai

Encoding wikitext-2-raw-v1 using OGA Tokenizer hangs

Open
#815 4 comments 0 reactions 1 assignee Claimed by @wenbingl View on GitHub
platform:windows
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
Using the OGA tokenizer to encode the wikitext-2-raw-v1 hangs and does not return, but works fine for wikitest-2-v1.

**To Reproduce**
Steps to reproduce the behavior:
import onnxruntime_genai as og
from datasets import load_dataset

tokenizer = og.Tokenizer(model)
testdata = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
tokenizer.encode("\n\n".join(testdata["text"]))

**Expected behavior**
Return a list of encoded wikitext2 dataset.

**Desktop (please complete the following information):**
- OS: Windows 11
- onnxruntime-genai-0.4.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.