How to add byte_fallback tokens?

# Alternative title

How to make a tokenizer behaving similarly to Llama

## Background

Llama tokenizer considers byte_fallback tokens **not special**. When it decodes, it doesn't remove these tokens other than special tokens (unk, pad, bos, eos).

## What I am trying to do

I'm trying to create a tokenizer behaving like Llama. However, I **am only able** to add byte_fallback tokens as **special tokens**.

```python
from tokenizers import Tokenizer
from tokenizers import decoders, pre_tokenizers
from tokenizers.models import BPE
from tokenizers.processors import TemplateProcessing
from tokenizers.trainers import BpeTrainer
from tokenizers import AddedToken

from datasets import load_dataset

dataset = load_dataset("tapaco")

def topaco_generator():
for i in dataset['train']:
yield i['paraphrase']

bpe_trainer = BpeTrainer(
special_tokens=["", "", "", ""]
+ [f"<0x{i:02X}>" for i in range(256)] # byte_fallback tokens
)

tokenizer = Tokenizer(BPE(byte_fallback=True))
tokenizer.pre_tokenizer = pre_tokenizers.Sequence(
[pre_tokenizers.Metaspace(), pre_tokenizers.Digits(individual_digits=True)]
)
tokenizer.enable_padding(pad_id=3, pad_token="")
tokenizer.post_processor = TemplateProcessing(
single=" $A ",
pair=" $A $B ",
special_tokens=[
("", 1),
("", 2),
],
)
tokenizer.decoder = decoders.Sequence(
[
decoders.Metaspace(),
decoders.ByteFallback(),
]
)
# my attempt to add byte_fallback as non-special tokens
# tokenizer.add_tokens([AddedToken(content=f"<0x{i:02X}>", special=True, normalized=False) for i in range(256)])

tokenizer.train_from_iterator(topaco_generator(), trainer=bpe_trainer)
tokenizer.save("topaco_tokenizer.json")

tokenizer = Tokenizer.from_file("topaco_tokenizer.json")

text = "I love you more than I can say 🤗"
encoded_text = tokenizer.encode(text)
print(encoded_text.tokens)
# My work around to preverse byte_fallback tokens
# and remove other special tokens
decoded_text = tokenizer.decode(encoded_text.ids, skip_special_tokens=False)
print(decoded_text.removeprefix(' ').removesuffix(''))
```

## Problem

No matter how I tried this line `tokenizer.add_tokens([AddedToken(content=f"<0x{i:02X}>", special=True, normalized=False) for i in range(256)])` with different position in my code (before training, after training) and with different parameters of AddedToken, I still can not achieve Llama's behavior.