ARBML / ARBML/tkseem

Model loading doesn't work for SentencePieceTokenizer

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
113
Forks
22
PR merge metrics
No merged PRs in 30d

Description

This code fails with an error:

```python
import tkseem as tk

tokenizer_path = 'model.pl'
tokenizer = tk.SentencePieceTokenizer()
tokenizer.train(dataset_file)

# save the tokenizer to a file
tokenizer.save_model(tokenizer_path)

# load the tokenizer from a file
tokenizer = tk.SentencePieceTokenizer()
tokenizer.load_model(tokenizer_path)

# test the tokenizer
a = tokenizer.tokenize("السلام عليكم")
```

Error message is:
```
Traceback (most recent call last):
File "/Users/user/Desktop/Projects/train-tokenizer.py", line 15, in
a = tokenizer.tokenize("السلام عليكم")
File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/site-packages/tkseem/sentencepiece_tokenizer.py", line 50, in tokenize
return self.sp.encode(text, out_type=str)
AttributeError: 'bool' object has no attribute 'encode'
```

The solution to this issue is updating the "load_model" to:
```python
def load_model(self, file_path):
"""Load a saved sp model

Args:
file_path (str): file path of the trained model
"""
self.sp = spm.SentencePieceProcessor(model_proto=open(file_path, "rb").read())
```

Contributor guide

No contributing guide indexed for this repository

Research direction

The traceback points to sentencepiece_tokenizer.py and the SentencePieceTokenizer.load_model entry point; start by reading that method and how tokenize uses self.sp. Confirm that loading model.pl leaves a usable processor and that tokenizing the Arabic example no longer raises the reported bool/encode error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.