huggingface / huggingface/tokenizers.js

Add offset mapping support for character-level token alignment

Open
#16 1 comment 0 reactions 0 assignees Claimed by @dkhokhlov View on GitHub
Dominant language
TypeScript
Stars
57
Forks
11
PR merge metrics
No merged PRs in 30d

Description

## Summary

Currently, the tokenizers.js library only provides token IDs, tokens, and attention masks in the `Encoding` interface. This lacks the character-level position information (`offset_mapping`) that's
available in the HuggingFace tokenizers Python/Rust library.

## Use Case

Character offset mapping is essential for:
- **NLP applications** that need to map tokens back to original text positions
- **PII detection models** (like NeuroBERT) where precise character locations are required
- **Text highlighting/annotation** based on tokenization results
- **Model interpretability** and token attribution
- **Compatibility** with HuggingFace tokenizers Python API

## Current Behavior

```javascript
const result = tokenizer.encode("Hello world!");
console.log(result);
// Output: {
// ids: [101, 7592, 2088, 999, 102],
// tokens: ["[CLS]", "hello", "world", "!", "[SEP]"],
// attention_mask: [1, 1, 1, 1, 1]
// // Missing: offset_mapping
// }

Expected Behavior

const result = tokenizer.encode("Hello world!", { return_offsets_mapping: true });
console.log(result.offset_mapping);
// Output: [[0,0], [0,5], [6,11], [11,12], [0,0]]
// [CLS] Hello world ! [SEP]

Requirements

- Add offset_mapping?: Array<[number, number]> to Encoding interface
- Add return_offsets_mapping?: boolean option to encode() method
- Implement character position tracking for WordPiece tokenizers
- Handle special tokens with [0,0] offsets (HF standard)
- Support subword tokens (## prefix) correctly
- Provide clear error messages for unsupported model types (BPE, Unigram)
- Maintain backward compatibility

Models to Support

Priority 1 (WordPiece):
- ✅ BERT, DistilBERT, ALBERT
- ✅ Medical/domain BERT variants (e.g., NeuroBERT, BioBERT)

Future (out of scope for initial implementation):
- BPE models (GPT, RoBERTa) - complex merge operations
- Unigram models (T5, XLM-R) - probabilistic segmentation

Additional Context

This feature would bring tokenizers.js closer to feature parity with the official HuggingFace tokenizers library and enable advanced NLP use cases that require precise character-level alignment.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.