AnswerDotAI / AnswerDotAI/ModernBERT
Training Reproducibility
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for putting together this awesome model!
I'm currently evaluating it in various benchmarks against other models (e.g., deberta-v3) using `AutoModelForSequenceClassification` for fine-tuning, and I'm having an issue with model reproducibility. Specifically, using existing code that allows for finding optimal hyperparameters and then subsequently refitting the final model, I am unable to make my ModernBERT model reproducible using coding that works fine for other model types (e.g., deberta-v3, roberta).
The process of refitting the model uses the huggingface recommended [`model_init`](https://discuss.huggingface.co/t/fixing-the-random-seed-in-the-trainer-does-not-produce-the-same-results-across-runs/3442/2) approach for reproducibility:
```python
final_trainer = Trainer(
args = final_args,
data_collator = data_collator,
model_init = model_init,
train_dataset = dataset_dict_tokenized['train'],
eval_dataset = dataset_dict_tokenized['val'],
compute_metrics = compute_metrics
)
```
However, while this works for all other models, it doesn't seem to for ModernBERT. Initially I thought this might be a flash-attn related issue, but it happens regardless of whether I use that or not.
Any advice here is much appreciated!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.