facebookresearch / facebookresearch/fairseq2

Feature request: make tokenizers map between characters and tokens

Open
#1,469 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe:**
The requested functionality is, with a text tokenizer (e.g. a BPE tokenizer like a LLama one), determine for each token the positions of the text characters corresponding to it.

For background: I need this for projecting token-level predictions from my text encoder back to spans of the original text.

In easy cases (pure ASCII characters), I can achieve this by applying `tokenizer.create_encoder(...).encode_as_tokens(text)` and then simply adding up the length of text-form tokens. However, with more difficult characters (e.g. non-Latin script), this method results in a `UnicodeDecodeError` if a token boundary happens to fall in the middle of a multi-byte character.

**Describe the solution you would like:**
An ideal solution would be either to implement a method like `token_to_chars` in HF tokenizers that does provide the requested mapping, or at least to publicly expose a method for converting tokens ids into byte sequences (currently, with TikToken-based tokenizers, I can get that with `tokenizer._encoding.decode_tokens_bytes(text_tokens)`, but this wouldn't work with other tokenizer types, and anyway, using a private property doesn't feel very good).

**Describe the alternatives you have considered:**
1. Using `tokenizer._encoding` to convert tokens into byte sequences and then manually align them with the text - this works but feels dirty.
2. Not using Fairseq2 tokenizers at all - this is alright, but I'd like to keep using them, if only for backward compatibility, in some well-established projects like SONAR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.