huggingface / huggingface/tokenizers
Access utf-8 byte sequence for each token
Open
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Hi,
It would be great if it was possible to get the utf-8 byte sequence corresponding to each token id.
Since tokenizers return strings, tokens which are not valid unicode strings by themselves will contain � on decode.
This eg. makes streaming and constrained generation much more difficult and error prone than it needs to be.
Additionally if we can get the uf8 byte sequence, decoding also get's much easier and faster, as it's simply a matter of concatenating the corresponding bytes.
Cheers,
Contributor guide
Assessment
This issue has not been assessed yet.