AnswerDotAI / AnswerDotAI/ModernBERT

Special tokens masking candidates

Open
#212 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Based on the masking implementation in the `transformers` library, special tokens (e.g., [CLS], [SEP]) should be excluded from the masking process. However, upon reviewing the implementation in sequence_packer.py,
https://github.com/AnswerDotAI/ModernBERT/blob/8c57a0f01c12c4953ead53d398a36f81a4ba9e38/src/sequence_packer.py#L284
it appears that these tokens are currently being treated as valid masking candidates.

Could you please confirm if this behavior is intentional? If not, I suggest updating the masking logic to explicitly exclude special tokens. For instance, adding a condition to filter out these tokens before applying the mask would ensure consistency with the `transformers` library's approach. Additionally, incorporating unit tests to verify that special tokens remain unmasked would improve code reliability.

Am I correct in my understanding, or is there something I might be missing?

Thank you for looking into this.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at src/sequence_packer.py around line 284 and compare candidate selection with the masking behavior in the transformers library. Confirm whether [CLS] and [SEP] are included, then add unit coverage showing special tokens remain unmasked if the behavior is unintended.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.