huggingface / huggingface/tokenizers
Making Rust custom components easily available in Python
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Hi! Thanks a lot for all the work put in this library!
I am interested in moving a custom pre-tokenizer I have created as a Python class via PyO3. Here is an example:
```rust
use tokenizers::tokenizer::{normalizer::Range, PreTokenizedString, PreTokenizer, Result};
use tokenizers::utils::macro_rules_attribute;
use tokenizers::impl_serde_type;
fn get_example_ranges(input: &str) -> Result> {
Ok(vec![(0, 2)])
}
#[derive(Clone, Debug, PartialEq, Eq)]
#[macro_rules_attribute(impl_serde_type!)]
pub struct CustomPreTokenizer;
impl Default for CustomPreTokenizer {
fn default() -> Self {
Self
}
}
impl PreTokenizer for CustomPreTokenizer {
fn pre_tokenize(&self, pretokenized: &mut PreTokenizedString) -> Result<()> {
pretokenized.split(|_, normalized| {
let ranges = get_example_ranges(normalized.get())?;
Ok(ranges
.into_iter()
.map(|item| {
normalized
.slice(Range::Normalized(item.0..item.1))
.expect("Invalid input")
})
.collect::>())
})
}
}
```
I am now interested in using this in a Python script, and after looking at the bindings code present in the library, I tried to implement the following:
```rust
#[pyclass(extends=PyPreTokenizer, name = "CustomPreTokenizer")]
pub struct PyCustomPreTokenizer {}
#[pymethods]
impl PyCustomPreTokenizer {
#[new]
#[pyo3(text_signature = "(self)")]
fn new() -> (Self, PyPreTokenizer) {
(PyCustomPreTokenizer {}, CustomPreTokenizer {}.into())
}
}
```
However, I could not seem to be able to import/call the PyPreTokenizer struct described in the following file:
https://github.com/huggingface/tokenizers/blob/fdd26ba9a3f0c133427aab0423888cbde91362d7/bindings/python/src/pre_tokenizers.rs#L38
Is there a way to achieve this without having to reimplement the functionality of the PyPreTokenizer struct in my project? For example, having a way to call `use tokenizers_pyo3::PyPreTokenizer`?
Contributor guide
Assessment
This issue has not been assessed yet.