huggingface / huggingface/tokenizers

Improve PySequence objects

Open
#659 3 comments 0 reactions 0 assignees View on GitHub
Feature Request Stale
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

With https://github.com/huggingface/tokenizers/pull/530 we made most of the components expose their attributes and allow their modification. We'd like to do the same for `PySequence` objects (both for `Normalizer` and `PreTokenizer` sequences).

In order to make these `PySequence` objects behave like a list, we'd need to implement the `PySequenceProtocol` with the `__len__`, `__getitem__` and `__setitem__` dunder methods. Unfortunately, these don't support using a `PyRef` at the moment (cf https://github.com/PyO3/pyo3/issues/1206), and we need it to access the content of the base class.

This is necessary to allow behavior such as:
```python
if isinstance(tokenizer.normalizer, Sequence):
# Access a normalizer in the sequence
print(tokenizer.normalizer[0])
# Modify a normalizer in the sequence
tokenizer.normalizer[1] = Lowercase()
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.