AnswerDotAI / AnswerDotAI/ModernBERT

`_pad_token` attribute?

Open
#241 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Thanks for making this open-source! The following function checks for `_pad_token` attribute:
```python
def _tokenize(self, text_sample):
if self.tokenizer._pad_token is None:
# Some tokenizers (e.g. GPT2 tokenizer) have no padding token which causes bugs
raise RuntimeError("If tokenizing on-the-fly, tokenizer must have a pad_token_id")

return self.tokenizer(text_sample["text"], truncation=True, padding="max_length", max_length=self.max_seq_len)
```

But shouldn't it simply check for `pad_token_id`? My tokenizer has `pad_token_id` and `pad_token`, but no `_pad_token`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the `_tokenize` method shown in the issue and inspect how the tokenizer’s padding attributes are used. Reproduce the case with a tokenizer that has `pad_token_id` and `pad_token` but no `_pad_token`, then check the behavior for GPT2-like tokenizers. Done means valid tokenizers can be tokenized while missing padding support still produces the stated error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.