huggingface / huggingface/tokenizers
Improve PySequence objects
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
With https://github.com/huggingface/tokenizers/pull/530 we made most of the components expose their attributes and allow their modification. We'd like to do the same for `PySequence` objects (both for `Normalizer` and `PreTokenizer` sequences).
In order to make these `PySequence` objects behave like a list, we'd need to implement the `PySequenceProtocol` with the `__len__`, `__getitem__` and `__setitem__` dunder methods. Unfortunately, these don't support using a `PyRef` at the moment (cf https://github.com/PyO3/pyo3/issues/1206), and we need it to access the content of the base class.
This is necessary to allow behavior such as:
```python
if isinstance(tokenizer.normalizer, Sequence):
# Access a normalizer in the sequence
print(tokenizer.normalizer[0])
# Modify a normalizer in the sequence
tokenizer.normalizer[1] = Lowercase()
```
Contributor guide
Assessment
This issue has not been assessed yet.