AnswerDotAI / AnswerDotAI/ModernBERT

Question about Different Model Loading Approaches for NLU vs Retrieval Tasks

Open
#154 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

### Current Behavior
I noticed two different approaches for loading ModernBERT in the examples:

1. For GLUE tasks ([finetune_modernbert_on_glue.ipynb](https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/finetune_modernbert_on_glue.ipynb)):
- Uses `AutoModelForSequenceClassification`
- Retains the pretrained `ModernBertPredictionHead`

2. For retrieval tasks ([train_st.py](https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/train_st.py)):
- Uses `SentenceTransformer(model_name)`
- Discards the pretrained `ModernBertPredictionHead`

### Question
Could you explain the design rationale behind using different model loading approaches for these tasks? I'm particularly interested in understanding why the prediction head is retained in one case but discarded in the other.

Thank you for maintaining this excellent project!

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare the model-loading sections in examples/finetune_modernbert_on_glue.ipynb and examples/train_st.py, then trace the referenced model classes and SentenceTransformer integration. Document the rationale for retaining or discarding the prediction head, with the explanation covering both GLUE and retrieval use cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.