AnswerDotAI/ModernBERT

View on GitHub

Bringing BERT into modernity via both architecture changes and scaling

Stars: 1.7k
Forks: 145
Open beginner issues: 0
Indexed issues: 55

Dominant language: Python
License: Apache-2.0
Last GitHub push: Mar 1, 2026
Latest indexed: Sep 14, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Open All

55 open issues indexed

Training Reproducibility Open

AnswerDotAI/ModernBERT#142 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
The checkpoint you are trying to load has model type `modernbert` but Transformers does not recognize this architecture Open

AnswerDotAI/ModernBERT#145 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
White space at the start of each token_str. Could this affect backwards compatibility perhaps? Feature or Bug? Open

AnswerDotAI/ModernBERT#146 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
ST train example doesn't work on multi gpu Open

AnswerDotAI/ModernBERT#147 · 8 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
On the performance of token classification Open

AnswerDotAI/ModernBERT#149 · 16 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
Inference fails on CPU: `ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)` Open

AnswerDotAI/ModernBERT#152 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
checkpoints for further pretraining Open

AnswerDotAI/ModernBERT#153 · 11 comments · 2 reactions · 1 assignee · Indexed Sep 14, 2026
Question about Different Model Loading Approaches for NLU vs Retrieval Tasks Open

AnswerDotAI/ModernBERT#154 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How to run pre-training? Open

AnswerDotAI/ModernBERT#155 · 7 comments · 14 reactions · 0 assignees · Indexed Sep 14, 2026
Evaluation Open

AnswerDotAI/ModernBERT#156 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Question about fine-tuning for text classification Open

AnswerDotAI/ModernBERT#157 · 6 comments · 3 reactions · 0 assignees · Indexed Sep 14, 2026
Vocab.txt for ONNX? Open

AnswerDotAI/ModernBERT#160 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Example of GreedyBestFitSequencePacker with HuggingFace Trainer Open

AnswerDotAI/ModernBERT#161 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
MaskedLM nan training loss Open

AnswerDotAI/ModernBERT#163 · 7 comments · 8 reactions · 0 assignees · Indexed Sep 14, 2026
Missing model_max_length for tokeniser Open

AnswerDotAI/ModernBERT#166 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
Does not support multiple gpu training. Also, please post how to convert your pretrain results into hugging faces. Open

AnswerDotAI/ModernBERT#169 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
What's the difference among the different mode when initialize model from pretrained model? Open

AnswerDotAI/ModernBERT#171 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
During pre-training, using FA2 consumes more memory than using SDPA Open

AnswerDotAI/ModernBERT#172 · 10 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Request for code to quantize and convert ModernBERT Model to ONNX Open

AnswerDotAI/ModernBERT#173 · 11 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
ModernBertModel works on the CPU but fails on the GPU Open

AnswerDotAI/ModernBERT#174 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Unable to reproduce Table 7 from the paper Open

AnswerDotAI/ModernBERT#176 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Yaml used to train the ModernBert Open

AnswerDotAI/ModernBERT#178 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 14, 2026
ModernBertPreTrainedModel for word-level prediction tasks Open

AnswerDotAI/ModernBERT#179 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
ModernBert Encoder Open

AnswerDotAI/ModernBERT#180 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Are there minimum hardware requirements regarding pre-training? Open

AnswerDotAI/ModernBERT#182 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
ImportError: cannot import name 'StreamingDataset' from 'streaming' Open

AnswerDotAI/ModernBERT#185 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How to Extract a Model from a Checkpoint Open

AnswerDotAI/ModernBERT#186 · 3 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
How to use a custom tokenizer for pre-training BERT from scratch Open

AnswerDotAI/ModernBERT#187 · 5 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
How to Tag Entities/Keywords in ModernBert? Open

AnswerDotAI/ModernBERT#188 · 0 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
TypeError: ModernBertForSequenceClassification.__init__() got an unexpected keyword argument 'compile' Open

AnswerDotAI/ModernBERT#190 · 6 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
contextual embeddings of individual words Open

AnswerDotAI/ModernBERT#191 · 5 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Can't find yamls/main/modernbert-base.yaml in repo Open

AnswerDotAI/ModernBERT#192 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Question about MLDR Evaluation Metrics in ModernBERT Paper Open

AnswerDotAI/ModernBERT#193 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 14, 2026
BackendCompilerFailed Open

AnswerDotAI/ModernBERT#198 · 4 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
`IndexError: tuple index out of range` in `input_ids` during pre-training Open

AnswerDotAI/ModernBERT#199 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
HuggingFace loss implementation Open

AnswerDotAI/ModernBERT#206 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Special tokens masking candidates Open

AnswerDotAI/ModernBERT#212 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 14, 2026
NaN rotaty_emb weights in exported pretrained models Open

AnswerDotAI/ModernBERT#215 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Is there a config file for ModernBERT-Large? Open

AnswerDotAI/ModernBERT#217 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Multi-GPU Pretraining is slower than single GPU training? Open

AnswerDotAI/ModernBERT#220 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How to continue to train ModernBert base/large in a specific domain? Open

AnswerDotAI/ModernBERT#222 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Training Tokenizer and pretraining Open

AnswerDotAI/ModernBERT#226 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Pretraining Throughput — Unexpected Drop in Speed Open

AnswerDotAI/ModernBERT#228 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
grad_norm Nan question Open

AnswerDotAI/ModernBERT#233 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Pretraining with FlexBert but can't export to HF Open

AnswerDotAI/ModernBERT#234 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
question to implementation for ModernBertForMaskedLM vs ModernBertForSequenceClassification Open

AnswerDotAI/ModernBERT#235 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
ValueError: Invalid pattern: '**' can only be an entire path component Open

AnswerDotAI/ModernBERT#237 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Usage of GTE-ModernColBERT or ModernBERT embedding model for storing file chunks and vector embeddings in a vector store Open

AnswerDotAI/ModernBERT#240 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
`_pad_token` attribute? Open

AnswerDotAI/ModernBERT#241 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Question about masking - why mask on unpadded sequence? Open

AnswerDotAI/ModernBERT#242 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026

AnswerDotAI/ModernBERT

Get new issues in your inbox