AnswerDotAI / AnswerDotAI/ModernBERT

Training Reproducibility

Open
#142 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Thank you for putting together this awesome model!

I'm currently evaluating it in various benchmarks against other models (e.g., deberta-v3) using `AutoModelForSequenceClassification` for fine-tuning, and I'm having an issue with model reproducibility. Specifically, using existing code that allows for finding optimal hyperparameters and then subsequently refitting the final model, I am unable to make my ModernBERT model reproducible using coding that works fine for other model types (e.g., deberta-v3, roberta).

The process of refitting the model uses the huggingface recommended [`model_init`](https://discuss.huggingface.co/t/fixing-the-random-seed-in-the-trainer-does-not-produce-the-same-results-across-runs/3442/2) approach for reproducibility:

```python
final_trainer = Trainer(
args = final_args,
data_collator = data_collator,
model_init = model_init,
train_dataset = dataset_dict_tokenized['train'],
eval_dataset = dataset_dict_tokenized['val'],
compute_metrics = compute_metrics
)
```

However, while this works for all other models, it doesn't seem to for ModernBERT. Initially I thought this might be a flash-attn related issue, but it happens regardless of whether I use that or not.

Any advice here is much appreciated!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.