dotnet / dotnet/machinelearning

[Tokenizers] BpeOptions.ByteLevel=true silently drops spaces and newlines when PreTokenizer is not set

Open
#7,715 1 comment 0 reactions 0 assignees View on GitHub
untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### Describe the bug

When a `BpeTokenizer` is created with `BpeOptions.ByteLevel = true` but **without** setting `BpeOptions.PreTokenizer`, the tokenizer silently drops spaces and newlines.

No exception is thrown and no warning is logged. `Decode` cannot recover the original text, and the resulting token IDs are *structurally valid but semantically wrong* — which makes the problem very hard to notice in a downstream LLM pipeline.

### To Reproduce

```csharp
// vocab : Dictionary parsed from tokenizer.json -> model.vocab
// merges : IEnumerable parsed from tokenizer.json -> model.merges
var options = new BpeOptions(vocab)
{
Merges = merges,
ByteLevel = true, // enables the byte <-> unicode mapping
UnknownToken = "",
FuseUnknownTokens = true,
// NOTE: PreTokenizer is intentionally NOT set here
};

var tokenizer = BpeTokenizer.Create(options);

string[] samples =
{
"Hello, world!",
"Tokenizer 是一个开源的中文分词器。",
"<|im_start|>system\nYou are a helpful assistant.<|im_end|>",
};

foreach (string s in samples)
{
IReadOnlyList ids = tokenizer.EncodeToIds(s);
string? back = tokenizer.Decode(ids);
Console.WriteLine($"in : {s}");
Console.WriteLine($"out: {back}");
Console.WriteLine(s == back ? "OK" : "LOST DATA");
}
```

**Actual result** — spaces and newlines are silently gone:

```
in : Hello, world!
out: Hello,world! <- LOST DATA (space)
in : Tokenizer 是一个开源的中文分词器。
out: Tokenizer是一个开源的中文分词器。 <- LOST DATA (space)
in : <|im_start|>system\nYou are a helpful assistant.<|im_end|>
out: <|im_start|>systemYou are a helpful assistant.<|im_end|> <- LOST DATA (newline)
```

Lossless round-trip rate with this configuration: **1 / 5**.

### Expected behavior

Enabling `ByteLevel` alone should not produce a tokenizer that corrupts its input. One of the following would be enough:

1. `BpeTokenizer.Create` throws (or logs a warning) when `ByteLevel == true` and `PreTokenizer == null`, or
2. the documentation for `BpeOptions.ByteLevel` states explicitly that a `PreTokenizer` must also be configured.

### Setting a PreTokenizer fixes it completely

With `RegexPreTokenizer` built from the GPT-2 pattern, every sample above round-trips losslessly:

```csharp
options.ByteLevel = true;
options.PreTokenizer = new RegexPreTokenizer(
new Regex(@"'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"),
specialTokens); // second argument: the added_tokens map
```

Result with `PreTokenizer` set: **6 / 6** samples match the HuggingFace reference token-for-token, **5 / 5** lossless round-trip.

### Measured configuration matrix

Reference implementation: HuggingFace `tokenizers` (Rust), loading the same `tokenizer.json`.

| `ByteLevel` | `PreTokenizer` | Match HF (6 samples) | Lossless round-trip |
|---|---|---|---|
| `true` | *(not set)* | 3 / 6 | **1 / 5** |
| `true` | `RobertaPreTokenizer` | 5 / 6 | — |
| `true` | `RegexPreTokenizer` (GPT-2 pattern) | **6 / 6** | **5 / 5** |
| `false` | `RobertaPreTokenizer` | 0 / 6 | — |
| `false` | `RegexPreTokenizer` (GPT-2 pattern) | 0 / 6 | — |

### Additional context — two documentation points

1. The current docs for `BpeOptions.ByteLevel` only describe the byte <-> unicode mapping (e.g. `Space -> 'Ġ'`). They do not mention that a `PreTokenizer` is required for correct whitespace handling.

2. The BPE sample in https://learn.microsoft.com/dotnet/ai/how-to/use-tokenizers contains this note:

> "BpeTokenizer might not always decode IDs to the exact original text as it can remove spaces during tokenization **depending on the model configuration**."

Based on the measurements above, the behaviour is **not** dependent on the model configuration. It depends solely on whether a `PreTokenizer` is configured. Once a proper pre-tokenizer is set, decoding is exact for every input tested.

### Why this matters for HuggingFace models

`tokenizer.json` carries the full pipeline configuration — `pre_tokenizer`, `added_tokens`, `normalizer`, `post_processor`, `decoder`. The `BpeTokenizer.Create(vocab, merges)` entry point consumes only the vocabulary data and cannot see any of it. Users following the current guidance (download `vocab.json` + `merges.txt` from HuggingFace) therefore end up with a tokenizer that is missing its pipeline configuration, and the failure mode is silent.

### Environment

- `Microsoft.ML.Tokenizers` 2.0.0 (stable)
- .NET 10
- OS: Windows

Contributor guide

Open the contributing guide

Research direction

Start at BpeTokenizer.Create and the BpeOptions.ByteLevel and PreTokenizer behavior described in the reproduction. Check the existing tokenizer tests or add coverage for encoding and decoding whitespace when ByteLevel is enabled without a pre-tokenizer. Done means the configuration no longer silently loses spaces and newlines, either by rejecting it clearly or documenting the required pre-tokenizer.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.