AnswerDotAI / AnswerDotAI/ModernBERT
Question about fine-tuning for text classification
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for the exceptional work! I have a query concerning the fine-tuning process for text classification.
I've been following the code provided in the 'examples/finetune_modernbert_on_glue.ipynb' notebook, but I've encountered some perplexing results. Could you please provide some guidance?

Here are more details:
1. I noticed that the var 'wd' wasn't being utilized, so I included it for weight decay, but this has led to confusing results too. You can see the screenshot of the error here:

2. I received a TypeError stating that ModernBertForSequenceClassification.__init__() received an unexpected keyword argument 'compile'. Here's the code snippet where the error occurred:
```python
hf_model = AutoModelForSequenceClassification.from_pretrained(
checkpoint, num_labels=n_labels, id2label=id2label, label2id=label2id, compile=False
)
```
Could you please provide guidance on how to address these issues? Thank you in advance for your help!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with examples/finetune_modernbert_on_glue.ipynb and reproduce the reported weight-decay and compile errors, including the AutoModelForSequenceClassification.from_pretrained call. Compare the notebook's handling of wd and model arguments, then confirm the documented fine-tuning path runs without those errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100