huggingface / huggingface/tokenizers

Making Rust custom components easily available in Python

Open
#1,566 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

Hi! Thanks a lot for all the work put in this library!

I am interested in moving a custom pre-tokenizer I have created as a Python class via PyO3. Here is an example:
```rust
use tokenizers::tokenizer::{normalizer::Range, PreTokenizedString, PreTokenizer, Result};
use tokenizers::utils::macro_rules_attribute;
use tokenizers::impl_serde_type;

fn get_example_ranges(input: &str) -> Result> {
Ok(vec![(0, 2)])
}

#[derive(Clone, Debug, PartialEq, Eq)]
#[macro_rules_attribute(impl_serde_type!)]
pub struct CustomPreTokenizer;

impl Default for CustomPreTokenizer {
fn default() -> Self {
Self
}
}

impl PreTokenizer for CustomPreTokenizer {
fn pre_tokenize(&self, pretokenized: &mut PreTokenizedString) -> Result<()> {
pretokenized.split(|_, normalized| {
let ranges = get_example_ranges(normalized.get())?;
Ok(ranges
.into_iter()
.map(|item| {
normalized
.slice(Range::Normalized(item.0..item.1))
.expect("Invalid input")
})
.collect::>())
})
}
}
```

I am now interested in using this in a Python script, and after looking at the bindings code present in the library, I tried to implement the following:
```rust
#[pyclass(extends=PyPreTokenizer, name = "CustomPreTokenizer")]
pub struct PyCustomPreTokenizer {}
#[pymethods]
impl PyCustomPreTokenizer {
#[new]
#[pyo3(text_signature = "(self)")]
fn new() -> (Self, PyPreTokenizer) {
(PyCustomPreTokenizer {}, CustomPreTokenizer {}.into())
}
}
```

However, I could not seem to be able to import/call the PyPreTokenizer struct described in the following file:
https://github.com/huggingface/tokenizers/blob/fdd26ba9a3f0c133427aab0423888cbde91362d7/bindings/python/src/pre_tokenizers.rs#L38

Is there a way to achieve this without having to reimplement the functionality of the PyPreTokenizer struct in my project? For example, having a way to call `use tokenizers_pyo3::PyPreTokenizer`?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.