AnswerDotAI / AnswerDotAI/ModernBERT
Question about Different Model Loading Approaches for NLU vs Retrieval Tasks
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
### Current Behavior
I noticed two different approaches for loading ModernBERT in the examples:
1. For GLUE tasks ([finetune_modernbert_on_glue.ipynb](https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/finetune_modernbert_on_glue.ipynb)):
- Uses `AutoModelForSequenceClassification`
- Retains the pretrained `ModernBertPredictionHead`
2. For retrieval tasks ([train_st.py](https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/train_st.py)):
- Uses `SentenceTransformer(model_name)`
- Discards the pretrained `ModernBertPredictionHead`
### Question
Could you explain the design rationale behind using different model loading approaches for these tasks? I'm particularly interested in understanding why the prediction head is retained in one case but discarded in the other.
Thank you for maintaining this excellent project!
Contributor guide
No contributing guide indexed for this repository
Research direction
Compare the model-loading sections in examples/finetune_modernbert_on_glue.ipynb and examples/train_st.py, then trace the referenced model classes and SentenceTransformer integration. Document the rationale for retaining or discarding the prediction head, with the explanation covering both GLUE and retrieval use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100