huggingface / huggingface/tokenizers

Access utf-8 byte sequence for each token

Open
#1,628 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

Hi,

It would be great if it was possible to get the utf-8 byte sequence corresponding to each token id.
Since tokenizers return strings, tokens which are not valid unicode strings by themselves will contain � on decode.

This eg. makes streaming and constrained generation much more difficult and error prone than it needs to be.

Additionally if we can get the uf8 byte sequence, decoding also get's much easier and faster, as it's simply a matter of concatenating the corresponding bytes.

Cheers,

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.