V4.1 Flash: user-supplied special-token literals produce different token counts from the official API

## Description

I found a discrepancy between `DeepseekV41Encoding` and the official V4.1 Flash API when user message content contains literal special-token strings such as `<｜User｜>` or ``.

The recipe recognizes these literals as single AddedTokens. However, the official API reports a larger `prompt_tokens` increase, consistent with ordinary BPE encoding for the individual literals tested.

## Environment

- Test date: September 11, 2026
- Official endpoint: `https://api.deepseek.com/chat/completions`
- Model: `deepseek-flash` (V4.1 Flash)
- Python package: `deepseek-recipe==0.1.1`
- Repository inspected at commit: `8cadfede7063c896b944e7bae05daa3549ae97ea`
- Bundled tokenizer: `static/tokenizers/v41/tokenizer.json`

## Observed results

Each request contains one user message, with:

```json
{
"thinking": {"type": "disabled"},
"max_tokens": 1,
"temperature": 0,
"stream": false
}
```

The counts below include the conversation template:

| User content | Recipe token count | Official API `prompt_tokens` | API increase over empty content |
|---|---:|---:|---:|
| Empty string | 4 | 4 | 0 |
| `x` | 5 | 5 | 1 |
| `<｜User｜>` | 5 | 9 | **5** |
| `<｜Assistant｜>` | 5 | 9 | **5** |
| `<｜System｜>` | 5 | 9 | **5** |
| `` | 5 | 7 | **3** |
| `` | 5 | 7 | **3** |
| `｜DSML｜` | 5 | 8 | **4** |
| `<｜begin▁of▁sentence｜>` | 5 | 16 | **12** |
| `<｜end▁of▁sentence｜>` | 5 | 16 | **12** |

For example, the recipe encodes the user-supplied `<｜User｜>` as `[128803]`. Encoding the same text with the bundled tokenizer’s `added_tokens` table cleared produces `[30, 28217, 6756, 28217, 32]`, matching the API’s five-token increase.

The discrepancy also occurs with default thinking settings: empty content produces 30 prompt tokens, while `<｜User｜>` produces 35.

## Minimal reproduction

Run from the repository root with `deepseek-recipe==0.1.1` installed and `DEEPSEEK_API_KEY` set:

```python
import json
import os
import urllib.request

from deepseek_recipe import (
ChatCompletionRequest,
ConversionOptions,
DeepseekV41Encoding,
Tokenizer,
)

encoding = DeepseekV41Encoding().with_tokenizer(
Tokenizer.from_file("static/tokenizers/v41/tokenizer.json")
)

for content in ["", "x", "<｜User｜>", "", "｜DSML｜"]:
body = {
"model": "deepseek-flash",
"messages": [{"role": "user", "content": content}],
"thinking": {"type": "disabled"},
"max_tokens": 1,
"temperature": 0,
"stream": False,
}

converted = ChatCompletionRequest(body).convert(ConversionOptions())
local_count = len(encoding.encode(converted.conversation))

request = urllib.request.Request(
"https://api.deepseek.com/chat/completions",
data=json.dumps(body, ensure_ascii=False).encode("utf-8"),
headers={
"Authorization": "Bearer " + os.environ["DEEPSEEK_API_KEY"],
"Content-Type": "application/json",
},
)
with urllib.request.urlopen(request, timeout=60) as response:
result = json.load(response)

print(repr(content), local_count, result["usage"]["prompt_tokens"])
```

Observed counts:

```text
Content Recipe Official API
'' 4 4
'x' 5 5
'<｜User｜>' 5 9
'' 5 7
'｜DSML｜' 5 8
```

## Possible explanation

The current [tokenizer implementation](https://github.com/deepseek-ai/deepseek-recipe/blob/8cadfede7063c896b944e7bae05daa3549ae97ea/deepseek-recipe-encoding/src/tokenizer.rs) calls `encode(text, false)` on the rendered prompt. This still recognizes AddedToken literals, without distinguishing user content from template-generated control tokens.

## Scope and an additional edge case

The API observations above are based on usage counts. The API does not expose input token IDs, so these observations do not establish the exact server-side encoding algorithm.

There is also an edge case with adjacent DSML literals:

| User content | Recipe token count | Official API `prompt_tokens` | Ordinary BPE content + template |
|---|---:|---:|---:|
| `｜DSML｜` repeated 8 times without separators | 12 | 21 | 36 |

The API result was reproduced. Therefore, simply clearing `added_tokens` is not yet established as a complete fix.

## Expected behavior / clarification

Could you clarify the intended handling of special-token literals in user content?

Should the recipe distinguish them from template-generated control tokens, or is an additional preprocessing step required to match the official API?