huggingface / huggingface/tokenizers

Bug: is_pretokenized is not used when calling tokenizer.encode(...)

Open
#1,695 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

is_pretokenized doesnt seem to be respected in some cases. The same code given below works in 0.20.0

## Code
```python
from tokenizers import Tokenizer, pre_tokenizer
from tokenizers.models import WordPiece

m = WordPiece({'F': 0, '': 1})
t = Tokenizer(m)
t.pre_tokenizer = pre_tokenizers.Split('', 'isolated')

t.encode([''], is_pretokenized=True).ids
```

Expected to run without any issue but raises the exception:
```
Exception: WordPiece error: Missing [UNK] token from the vocabulary
```

It seems to ignore the `is_pretokenized` flag and wants to apply the pre_tokenizer to the `` token.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.