microsoft / microsoft/onnxruntime-genai
Encoding wikitext-2-raw-v1 using OGA Tokenizer hangs
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
**Describe the bug**
Using the OGA tokenizer to encode the wikitext-2-raw-v1 hangs and does not return, but works fine for wikitest-2-v1.
**To Reproduce**
Steps to reproduce the behavior:
import onnxruntime_genai as og
from datasets import load_dataset
tokenizer = og.Tokenizer(model)
testdata = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
tokenizer.encode("\n\n".join(testdata["text"]))
**Expected behavior**
Return a list of encoded wikitext2 dataset.
**Desktop (please complete the following information):**
- OS: Windows 11
- onnxruntime-genai-0.4.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.