Model loading doesn't work for SentencePieceTokenizer
- Dominant language
- Jupyter Notebook
- Stars
- 113
- Forks
- 22
- PR merge metrics
- No merged PRs in 30d
Description
This code fails with an error:
```python
import tkseem as tk
tokenizer_path = 'model.pl'
tokenizer = tk.SentencePieceTokenizer()
tokenizer.train(dataset_file)
# save the tokenizer to a file
tokenizer.save_model(tokenizer_path)
# load the tokenizer from a file
tokenizer = tk.SentencePieceTokenizer()
tokenizer.load_model(tokenizer_path)
# test the tokenizer
a = tokenizer.tokenize("السلام عليكم")
```
Error message is:
```
Traceback (most recent call last):
File "/Users/user/Desktop/Projects/train-tokenizer.py", line 15, in
a = tokenizer.tokenize("السلام عليكم")
File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/site-packages/tkseem/sentencepiece_tokenizer.py", line 50, in tokenize
return self.sp.encode(text, out_type=str)
AttributeError: 'bool' object has no attribute 'encode'
```
The solution to this issue is updating the "load_model" to:
```python
def load_model(self, file_path):
"""Load a saved sp model
Args:
file_path (str): file path of the trained model
"""
self.sp = spm.SentencePieceProcessor(model_proto=open(file_path, "rb").read())
```
Contributor guide
No contributing guide indexed for this repository
Research direction
The traceback points to sentencepiece_tokenizer.py and the SentencePieceTokenizer.load_model entry point; start by reading that method and how tokenize uses self.sp. Confirm that loading model.pl leaves a usable processor and that tokenizing the Arabic example no longer raises the reported bool/encode error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100