huggingface / huggingface/tokenizers

Cannot inject custom PreTokenizer into Tokenizer

Open
#1,634 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

Hey,

I want to train a Tokenizer that operates on a custom PreTokenizer. I tried a mix of [this documentation post](https://huggingface.co/docs/tokenizers/pipeline) and [this example](https://github.com/huggingface/tokenizers/blob/b24a2fc1781d5da4e6ebcd3ecb5b91edffc0a05f/bindings/python/examples/custom_components.py). My resulting code looks like this:

```python
class GlyLESPreTokenizer:
def __init__(self, *args, **kwargs):
pass

def __new__(cls, *args, **kwargs):
return super().__new__(cls)

def glyles_split(self, iupac: str):
iuapc = iupac.strip().replace(" ", "")
token = CommonTokenStream(GlyLESLexer(InputStream(data="{" + iupac + "}")))
GlyLESParser(token).start()
idx = 0
output = []
for i in range(1, len(token.tokens) - 2):
txt = str(token.tokens[i].text)
output.append((txt, (idx, idx + len(txt))))
idx += len(txt)
return output

def pre_tokenize_str(self, input_: str):
return self.glyles_split(input_)

iupac = "QuiNAlaAc(b1-4)GalNAcA(a1-4)GalOAc(a1-2)QuiNAlaAc"

# This returns a list of 33 token
GlyLESPreTokenizer().pre_tokenize_str(iupac)

# This however only returns a list with one token that is the entire input string.
pre_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()).pre_tokenize_str(iupac)
```

The final idea is to use it in such setting:
```python
tokenizer = Tokenizer(models.Model())
tokenizer.normalizer = normalizers.Strip()
tokenizer.pre_tokenizer = pre_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer())
```

Can someone help me to understand how to use the `pre_tokenizers.PreTokenizer.custom` method to inject a custom, python-written PreTokenizer into a Tokenizer? Unfortunately, it is far beyond the scope of the project to convert the logic from [GlyLES](https://github.com/kalininalab/GlyLES) to RUST, so it has to be a Python PreTokenizer-class that is somehow injected into the Tokenizer.

Thank you for any help, comment, or feedback in advance.
Roman

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.