huggingface / huggingface/transfer-learning-conv-ai
add_prefix_space missing from tokenization
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 430
- PR merge metrics
- No merged PRs in 30d
Description
In your docs, you mention that whenever the GPT2 tokenizer is used, there should be a space prefixed to the input string: https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer
However, in the `get_dataset()` function in `utils.py`, you do not add `add_prefix_space=True` to the tokenizer call. Why is this the case?
Thanks for this great toolkit!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with get_dataset() in utils.py and compare its tokenizer call with the GPT2 guidance in the linked documentation. Determine whether the missing add_prefix_space=True is intentional; done means either updating the call or documenting why it should remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100