huggingface / huggingface/tokenizers

NormalizedString.clear() broken?

Open
#1,636 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

Hello. I think there are some problems with `NormalizedString` (tokenizers 0.15.2).

In the following example, `append()` works as expected.
```
from tokenizers import NormalizedString

s = NormalizedString("Hi.") # NormalizedString(original="Hi.", normalized="Hi.")
s.append("Hello.") # NormalizedString(original="Hi.", normalized="Hi. Hello.")
```

After using `clear()`, `append()` no longer modifies the `normalized` attribute.
```
from tokenizers import NormalizedString

s = NormalizedString("Hi.") # NormalizedString(original="Hi.", normalized="Hi.")
s.clear() # NormalizedString(original="Hi.", normalized="")
s.append("Hello.") # NormalizedString(original="Hi.", normalized="")
```

This is also a problem with `prepend`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.