AnswerDotAI / AnswerDotAI/ModernBERT

Question about fine-tuning for text classification

Open
#157 6 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Thank you for the exceptional work! I have a query concerning the fine-tuning process for text classification.
I've been following the code provided in the 'examples/finetune_modernbert_on_glue.ipynb' notebook, but I've encountered some perplexing results. Could you please provide some guidance?
![屏幕截图 2024-12-23 235715](https://github.com/user-attachments/assets/182f2e85-47ac-4e1f-8637-b4529922cf63)

Here are more details:
1. I noticed that the var 'wd' wasn't being utilized, so I included it for weight decay, but this has led to confusing results too. You can see the screenshot of the error here:
![屏幕截图 2024-12-24 000556](https://github.com/user-attachments/assets/51f7b0c5-cb15-4d83-958d-b9eca2287a57)
2. I received a TypeError stating that ModernBertForSequenceClassification.__init__() received an unexpected keyword argument 'compile'. Here's the code snippet where the error occurred:
```python
hf_model = AutoModelForSequenceClassification.from_pretrained(
checkpoint, num_labels=n_labels, id2label=id2label, label2id=label2id, compile=False
)
```
Could you please provide guidance on how to address these issues? Thank you in advance for your help!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with examples/finetune_modernbert_on_glue.ipynb and reproduce the reported weight-decay and compile errors, including the AutoModelForSequenceClassification.from_pretrained call. Compare the notebook's handling of wd and model arguments, then confirm the documented fine-tuning path runs without those errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.