AnswerDotAI / AnswerDotAI/ModernBERT
`_pad_token` attribute?
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Thanks for making this open-source! The following function checks for `_pad_token` attribute:
```python
def _tokenize(self, text_sample):
if self.tokenizer._pad_token is None:
# Some tokenizers (e.g. GPT2 tokenizer) have no padding token which causes bugs
raise RuntimeError("If tokenizing on-the-fly, tokenizer must have a pad_token_id")
return self.tokenizer(text_sample["text"], truncation=True, padding="max_length", max_length=self.max_seq_len)
```
But shouldn't it simply check for `pad_token_id`? My tokenizer has `pad_token_id` and `pad_token`, but no `_pad_token`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the `_tokenize` method shown in the issue and inspect how the tokenizer’s padding attributes are used. Reproduce the case with a tokenizer that has `pad_token_id` and `pad_token` but no `_pad_token`, then check the behavior for GPT2-like tokenizers. Done means valid tokenizers can be tokenized while missing padding support still produces the stated error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100