AnswerDotAI / AnswerDotAI/ModernBERT

NaN rotaty_emb weights in exported pretrained models

Open
#215 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

I used `pretraining_documentation/convert_to_hf.py` to convert pretrain checkpoint pt to HF models, as some model variants don't need to extend context window, but I found that output logits are nan, and `rotaty_emb` is nan in local attention layers

The fix is simple, just change `config.json` content:

* old: `local_rope_theta: -1`
* fixed: `local_rope_theta: 10000.0`, the same as global and as in training

and reload model from dir again, then model outputs totally make sense.

I guess this is because pretrain yamls don't have this value, and FlexBERTConfig [defaults to -1](https://github.com/AnswerDotAI/ModernBERT/blob/8c57a0f01c12c4953ead53d398a36f81a4ba9e38/src/bert_layers/configuration_bert.py#L94), but I'm not quite sure what would be a proper fix.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pretraining_documentation/convert_to_hf.py and inspect how the exported config.json gets its local_rope_theta value. Compare that path with FlexBERTConfig in src/bert_layers/configuration_bert.py and the pretraining YAMLs; done means converted local attention rotary embeddings and model logits are no longer NaN without manually editing config.json.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.