AnswerDotAI / AnswerDotAI/ModernBERT

White space at the start of each token_str. Could this affect backwards compatibility perhaps? Feature or Bug?

Open
#146 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

I just happened to realize that calls to BERT would output a `token_str = "word"` whereas a similar call to modernBERT outputs `token_str = " word"` with an additional white space before the `token_str`. Could this create some compatibility issues for people who want to swop in modernBERT to an existing workflow?

```
import torch
from transformers import pipeline
from pprint import pprint

input_text = "COVID is a [MASK]."

pipe = pipeline(
"fill-mask",
model="answerdotai/ModernBERT-base",
torch_dtype=torch.bfloat16,
)

results = pipe(input_text)
pprint(results)

pipe = pipeline(
"fill-mask",
model="bert-base-uncased",
torch_dtype=torch.bfloat16,
)
results = pipe(input_text)
pprint(results)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the reported difference with the transformers fill-mask pipeline using ModernBERT-base and bert-base-uncased, then inspect how each model produces token_str. Determine whether the leading whitespace is intentional or a compatibility bug; done means the behavior is explained and any required change has a regression test or documented rationale.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.