huggingface / huggingface/transfer-learning-conv-ai

add_prefix_space missing from tokenization

Open
#67 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.8k
Forks
430
PR merge metrics
No merged PRs in 30d

Description

In your docs, you mention that whenever the GPT2 tokenizer is used, there should be a space prefixed to the input string: https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer

However, in the `get_dataset()` function in `utils.py`, you do not add `add_prefix_space=True` to the tokenizer call. Why is this the case?

Thanks for this great toolkit!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with get_dataset() in utils.py and compare its tokenizer call with the GPT2 guidance in the linked documentation. Determine whether the missing add_prefix_space=True is intentional; done means either updating the call or documenting why it should remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.