huggingface / huggingface/tokenizers
Cannot inject custom PreTokenizer into Tokenizer
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Hey,
I want to train a Tokenizer that operates on a custom PreTokenizer. I tried a mix of [this documentation post](https://huggingface.co/docs/tokenizers/pipeline) and [this example](https://github.com/huggingface/tokenizers/blob/b24a2fc1781d5da4e6ebcd3ecb5b91edffc0a05f/bindings/python/examples/custom_components.py). My resulting code looks like this:
```python
class GlyLESPreTokenizer:
def __init__(self, *args, **kwargs):
pass
def __new__(cls, *args, **kwargs):
return super().__new__(cls)
def glyles_split(self, iupac: str):
iuapc = iupac.strip().replace(" ", "")
token = CommonTokenStream(GlyLESLexer(InputStream(data="{" + iupac + "}")))
GlyLESParser(token).start()
idx = 0
output = []
for i in range(1, len(token.tokens) - 2):
txt = str(token.tokens[i].text)
output.append((txt, (idx, idx + len(txt))))
idx += len(txt)
return output
def pre_tokenize_str(self, input_: str):
return self.glyles_split(input_)
iupac = "QuiNAlaAc(b1-4)GalNAcA(a1-4)GalOAc(a1-2)QuiNAlaAc"
# This returns a list of 33 token
GlyLESPreTokenizer().pre_tokenize_str(iupac)
# This however only returns a list with one token that is the entire input string.
pre_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()).pre_tokenize_str(iupac)
```
The final idea is to use it in such setting:
```python
tokenizer = Tokenizer(models.Model())
tokenizer.normalizer = normalizers.Strip()
tokenizer.pre_tokenizer = pre_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer())
```
Can someone help me to understand how to use the `pre_tokenizers.PreTokenizer.custom` method to inject a custom, python-written PreTokenizer into a Tokenizer? Unfortunately, it is far beyond the scope of the project to convert the logic from [GlyLES](https://github.com/kalininalab/GlyLES) to RUST, so it has to be a Python PreTokenizer-class that is somehow injected into the Tokenizer.
Thank you for any help, comment, or feedback in advance.
Roman
Contributor guide
Assessment
This issue has not been assessed yet.