dotnet / dotnet/machinelearning
[Tokenizers] BpeOptions.ByteLevel=true silently drops spaces and newlines when PreTokenizer is not set
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
### Describe the bug
When a `BpeTokenizer` is created with `BpeOptions.ByteLevel = true` but **without** setting `BpeOptions.PreTokenizer`, the tokenizer silently drops spaces and newlines.
No exception is thrown and no warning is logged. `Decode` cannot recover the original text, and the resulting token IDs are *structurally valid but semantically wrong* — which makes the problem very hard to notice in a downstream LLM pipeline.
### To Reproduce
```csharp
// vocab : Dictionary parsed from tokenizer.json -> model.vocab
// merges : IEnumerable parsed from tokenizer.json -> model.merges
var options = new BpeOptions(vocab)
{
Merges = merges,
ByteLevel = true, // enables the byte <-> unicode mapping
UnknownToken = "",
FuseUnknownTokens = true,
// NOTE: PreTokenizer is intentionally NOT set here
};
var tokenizer = BpeTokenizer.Create(options);
string[] samples =
{
"Hello, world!",
"Tokenizer 是一个开源的中文分词器。",
"<|im_start|>system\nYou are a helpful assistant.<|im_end|>",
};
foreach (string s in samples)
{
IReadOnlyList ids = tokenizer.EncodeToIds(s);
string? back = tokenizer.Decode(ids);
Console.WriteLine($"in : {s}");
Console.WriteLine($"out: {back}");
Console.WriteLine(s == back ? "OK" : "LOST DATA");
}
```
**Actual result** — spaces and newlines are silently gone:
```
in : Hello, world!
out: Hello,world! <- LOST DATA (space)
in : Tokenizer 是一个开源的中文分词器。
out: Tokenizer是一个开源的中文分词器。 <- LOST DATA (space)
in : <|im_start|>system\nYou are a helpful assistant.<|im_end|>
out: <|im_start|>systemYou are a helpful assistant.<|im_end|> <- LOST DATA (newline)
```
Lossless round-trip rate with this configuration: **1 / 5**.
### Expected behavior
Enabling `ByteLevel` alone should not produce a tokenizer that corrupts its input. One of the following would be enough:
1. `BpeTokenizer.Create` throws (or logs a warning) when `ByteLevel == true` and `PreTokenizer == null`, or
2. the documentation for `BpeOptions.ByteLevel` states explicitly that a `PreTokenizer` must also be configured.
### Setting a PreTokenizer fixes it completely
With `RegexPreTokenizer` built from the GPT-2 pattern, every sample above round-trips losslessly:
```csharp
options.ByteLevel = true;
options.PreTokenizer = new RegexPreTokenizer(
new Regex(@"'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"),
specialTokens); // second argument: the added_tokens map
```
Result with `PreTokenizer` set: **6 / 6** samples match the HuggingFace reference token-for-token, **5 / 5** lossless round-trip.
### Measured configuration matrix
Reference implementation: HuggingFace `tokenizers` (Rust), loading the same `tokenizer.json`.
| `ByteLevel` | `PreTokenizer` | Match HF (6 samples) | Lossless round-trip |
|---|---|---|---|
| `true` | *(not set)* | 3 / 6 | **1 / 5** |
| `true` | `RobertaPreTokenizer` | 5 / 6 | — |
| `true` | `RegexPreTokenizer` (GPT-2 pattern) | **6 / 6** | **5 / 5** |
| `false` | `RobertaPreTokenizer` | 0 / 6 | — |
| `false` | `RegexPreTokenizer` (GPT-2 pattern) | 0 / 6 | — |
### Additional context — two documentation points
1. The current docs for `BpeOptions.ByteLevel` only describe the byte <-> unicode mapping (e.g. `Space -> 'Ġ'`). They do not mention that a `PreTokenizer` is required for correct whitespace handling.
2. The BPE sample in https://learn.microsoft.com/dotnet/ai/how-to/use-tokenizers contains this note:
> "BpeTokenizer might not always decode IDs to the exact original text as it can remove spaces during tokenization **depending on the model configuration**."
Based on the measurements above, the behaviour is **not** dependent on the model configuration. It depends solely on whether a `PreTokenizer` is configured. Once a proper pre-tokenizer is set, decoding is exact for every input tested.
### Why this matters for HuggingFace models
`tokenizer.json` carries the full pipeline configuration — `pre_tokenizer`, `added_tokens`, `normalizer`, `post_processor`, `decoder`. The `BpeTokenizer.Create(vocab, merges)` entry point consumes only the vocabulary data and cannot see any of it. Users following the current guidance (download `vocab.json` + `merges.txt` from HuggingFace) therefore end up with a tokenizer that is missing its pipeline configuration, and the failure mode is silent.
### Environment
- `Microsoft.ML.Tokenizers` 2.0.0 (stable)
- .NET 10
- OS: Windows
Contributor guide
Research direction
Start at BpeTokenizer.Create and the BpeOptions.ByteLevel and PreTokenizer behavior described in the reproduction. Check the existing tokenizer tests or add coverage for encoding and decoding whitespace when ByteLevel is enabled without a pre-tokenizer. Done means the configuration no longer silently loses spaces and newlines, either by rejecting it clearly or documenting the required pre-tokenizer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100