huggingface / huggingface/course
Error in Training a Byte-Level BPE in "Building a tokenizer, block by block"
- Dominant language
- MDX
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
This [line](https://github.com/huggingface/course/blob/17d11ca412d5a303d98c90166ab3c895ff89e4ef/chapters/en/chapter6/8.mdx?plain=1#L342) for training a Byte-Level BPE has an error. You have to add an initial [alphabet of bytes](https://github.com/huggingface/tokenizers/blob/e7ed39de3c197492971d3b9076139d43a8ecdacb/bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py#L116), otherwise the tokenizer will not fall back to bytes when tokens are missing from the vocabulary and characters from your string can be missing when decoded.
For good reference and helping people, the training of a Byte-Level BPE should go as in [this example](https://github.com/huggingface/tokenizers/blob/main/bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py#L116).
Here is some shortened code so you don't have to follow the link or read it in a broken up tutorial:
```python
tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=trim_offsets)
def batch_generator(ds, batch_size):
for i in range(0, len(ds), batch_size):
yield ds[i : i + batch_size]['text']
trainer = trainers.BpeTrainer(
vocab_size=vocab_size,
min_frequency=min_frequency,
show_progress=show_progress,
special_tokens=special_tokens,
initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
)
self._tokenizer.train_from_iterator(
batch_generator,
trainer=trainer,
length=length,
)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.