huggingface / huggingface/course

Error in Training a Byte-Level BPE in "Building a tokenizer, block by block"

Open
#775 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
4.2k
Forks
1.4k
Avg merge
13m
Merged PRs (30d)
1

Description

This [line](https://github.com/huggingface/course/blob/17d11ca412d5a303d98c90166ab3c895ff89e4ef/chapters/en/chapter6/8.mdx?plain=1#L342) for training a Byte-Level BPE has an error. You have to add an initial [alphabet of bytes](https://github.com/huggingface/tokenizers/blob/e7ed39de3c197492971d3b9076139d43a8ecdacb/bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py#L116), otherwise the tokenizer will not fall back to bytes when tokens are missing from the vocabulary and characters from your string can be missing when decoded.

For good reference and helping people, the training of a Byte-Level BPE should go as in [this example](https://github.com/huggingface/tokenizers/blob/main/bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py#L116).

Here is some shortened code so you don't have to follow the link or read it in a broken up tutorial:
```python
tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=trim_offsets)

def batch_generator(ds, batch_size):
for i in range(0, len(ds), batch_size):
yield ds[i : i + batch_size]['text']

trainer = trainers.BpeTrainer(
vocab_size=vocab_size,
min_frequency=min_frequency,
show_progress=show_progress,
special_tokens=special_tokens,
initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
)
self._tokenizer.train_from_iterator(
batch_generator,
trainer=trainer,
length=length,
)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.