huggingface / huggingface/swift-transformers

Add offset mapping support to tokenizer.encode()

Open
#307 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Swift
Stars
1.4k
Forks
209
Avg merge
2d 7h
Merged PRs (30d)
4

Description

### Problem
When using BERT-based models for Named Entity Recognition (NER) tasks, I need to map token indices back to character positions in the
original text for entity extraction. Currently, there is no public API to obtain character offsets for tokens.

### Use Case
A common NER workflow:
1. Tokenize text: "John Smith works at Google"
2. Run NER model → get entity predictions with token indices
3. Need offsets to map tokens back to text → extract "John Smith" at positions [0-10]

### Current Limitation
The `encode()` and `tokenize()` methods only return token IDs and token strings, not their character positions in the original text. While `TokenLatticeNode` internally tracks `startOffset` and `length`, this information is not exposed publicly.
**Example :**
```
// Current API - no offset information
let ids = tokenizer.encode(text: "John Smith works at Google")
// Returns: [101, 2054, 3smith, 3638, 1012, 102]
// But we need: token positions in the original text
```

### Proposed Solution
Add a new method to the `Tokenizer` protocol that returns token offsets:
```
/// Result of encoding text with offset information
public struct EncodingWithOffsets {
public let ids: [Int] // Token IDs
public let tokens: [String] // Token strings
public let offsets: [(Int, Int)] // [(start, end), ...] character positions
}

// New method in Tokenizer protocol
public func encodeWithOffsets(text: String, addSpecialTokens: Bool = true) -> EncodingWithOffsets
```

### Benefits
- Consistent with Hugging Face Python API (return_offsets_mapping=True)
- No breaking changes to existing API (new method, not modification)

### Related Issues
This aligns with the tokenizer implementation pattern from the [Hugging Face transformers library](https://huggingface.co/docs/transformers/en/glossary#offsets-mapping) which has `offset_mapping` as a standard feature.
[Other example](https://discuss.huggingface.co/t/return-offsets-mapping-when-decoding/152215)

PS: First time for me to open an issue, so sorry if that not fit correct format. And thank you for this repo, he helps me a lot to start in ML.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.