AnswerDotAI / AnswerDotAI/ModernBERT
White space at the start of each token_str. Could this affect backwards compatibility perhaps? Feature or Bug?
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
I just happened to realize that calls to BERT would output a `token_str = "word"` whereas a similar call to modernBERT outputs `token_str = " word"` with an additional white space before the `token_str`. Could this create some compatibility issues for people who want to swop in modernBERT to an existing workflow?
```
import torch
from transformers import pipeline
from pprint import pprint
input_text = "COVID is a [MASK]."
pipe = pipeline(
"fill-mask",
model="answerdotai/ModernBERT-base",
torch_dtype=torch.bfloat16,
)
results = pipe(input_text)
pprint(results)
pipe = pipeline(
"fill-mask",
model="bert-base-uncased",
torch_dtype=torch.bfloat16,
)
results = pipe(input_text)
pprint(results)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the reported difference with the transformers fill-mask pipeline using ModernBERT-base and bert-base-uncased, then inspect how each model produces token_str. Determine whether the leading whitespace is intentional or a compatibility bug; done means the behavior is explained and any required change has a regression test or documented rationale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100