huggingface / huggingface/tokenizers
Issue with `SentencePieceUnigramTokenizer` Handling Unknown Tokens
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
**Description:**
When using the `SentencePieceUnigramTokenizer` with a custom vocabulary, there is no attribute to handle the `unk_id`, causing errors when encoding text not present in the vocabulary.
**Example:**
- **Vocabulary:** `{'a': -1.23, 'b': -1.34, 'c': -1.45}`
- **Encoding:** tokenizer.encode("bcd")

**Suggested Fix:**
```python
class SentencePieceUnigramTokenizer(BaseTokenizer):
def __init__(
self,
vocab: Optional[List[Tuple[str, float]]] = None,
replacement: str = "▁",
add_prefix_space: bool = True,
unk_id: int = 0,
):
if vocab is not None:
tokenizer = Tokenizer(Unigram(vocab, unk_id=unk_id))
else:
tokenizer = Tokenizer(Unigram())
tokenizer.normalizer = normalizers.Sequence([
normalizers.Nmt(),
normalizers.NFKC(),
normalizers.Replace(Regex(" {2,}"), " "),
])
tokenizer.pre_tokenizer = pre_tokenizers.Metaspace(replacement=replacement)
tokenizer.decoder = decoders.Metaspace(replacement=replacement)
parameters = {
"model": "SentencePieceUnigram",
"replacement": replacement,
"add_prefix_space": add_prefix_space,
}
super().__init__(tokenizer, parameters)
```
**Issue:**
The current implementation does not handle the `unk_id`, leading to errors when encountering unknown tokens. Adding support for `unk_id` in the tokenizer initialization would resolve this issue.
Please let me know is there any existing solution to it.
Contributor guide
Assessment
This issue has not been assessed yet.