huggingface / huggingface/tokenizers

Issue with `SentencePieceUnigramTokenizer` Handling Unknown Tokens

Open
#1,576 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

**Description:**

When using the `SentencePieceUnigramTokenizer` with a custom vocabulary, there is no attribute to handle the `unk_id`, causing errors when encoding text not present in the vocabulary.

**Example:**

- **Vocabulary:** `{'a': -1.23, 'b': -1.34, 'c': -1.45}`
- **Encoding:** tokenizer.encode("bcd")
![image](https://github.com/user-attachments/assets/2622c12d-8c15-4933-a1c0-9a9bea7b4b28)

**Suggested Fix:**

```python
class SentencePieceUnigramTokenizer(BaseTokenizer):
def __init__(
self,
vocab: Optional[List[Tuple[str, float]]] = None,
replacement: str = "▁",
add_prefix_space: bool = True,
unk_id: int = 0,
):
if vocab is not None:
tokenizer = Tokenizer(Unigram(vocab, unk_id=unk_id))
else:
tokenizer = Tokenizer(Unigram())

tokenizer.normalizer = normalizers.Sequence([
normalizers.Nmt(),
normalizers.NFKC(),
normalizers.Replace(Regex(" {2,}"), " "),
])
tokenizer.pre_tokenizer = pre_tokenizers.Metaspace(replacement=replacement)
tokenizer.decoder = decoders.Metaspace(replacement=replacement)

parameters = {
"model": "SentencePieceUnigram",
"replacement": replacement,
"add_prefix_space": add_prefix_space,
}

super().__init__(tokenizer, parameters)
```

**Issue:**

The current implementation does not handle the `unk_id`, leading to errors when encountering unknown tokens. Adding support for `unk_id` in the tokenizer initialization would resolve this issue.

Please let me know is there any existing solution to it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.