huggingface / huggingface/tokenizers
NormalizedString.clear() broken?
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Hello. I think there are some problems with `NormalizedString` (tokenizers 0.15.2).
In the following example, `append()` works as expected.
```
from tokenizers import NormalizedString
s = NormalizedString("Hi.") # NormalizedString(original="Hi.", normalized="Hi.")
s.append("Hello.") # NormalizedString(original="Hi.", normalized="Hi. Hello.")
```
After using `clear()`, `append()` no longer modifies the `normalized` attribute.
```
from tokenizers import NormalizedString
s = NormalizedString("Hi.") # NormalizedString(original="Hi.", normalized="Hi.")
s.clear() # NormalizedString(original="Hi.", normalized="")
s.append("Hello.") # NormalizedString(original="Hi.", normalized="")
```
This is also a problem with `prepend`.
Contributor guide
Assessment
This issue has not been assessed yet.