ModernBertPreTrainedModel for word-level prediction tasks

Hi there! I am trying to fine-tune ModernBertPreTrainedModel to complete the task of predicting each word in a sentence.

I customized a model, but when the model class inherits ModernBertPreTrainedModel and uses the from_pretrained method to load the model, I found that the performance is completely different from when the model class directly inherits nn.Module for training (the latter is better).

However, this problem was not found on other categories of Bert based models on same task. This troubled me for some time, and I still can't figure out why this situation happened.

Thanks!

Start by reproducing the word-level prediction task with the custom class inheriting ModernBertPreTrainedModel, comparing it with the nn.Module version. Inspect how from_pretrained is used in the reported setup and determine why the training performance differs from other BERT-based models. Done means identifying a reproducible cause and documenting or correcting the discrepancy.

Written by the indexing model from the issue text.