AnswerDotAI / AnswerDotAI/ModernBERT

On the performance of token classification

Open
#149 16 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Hi guys,

I spent some time evaluating ModernBERT Large on the standard CoNLL-2003 shared task.

For fine-tuning I used the following [code](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) to fine-tune:

```bash
python3 run_ner.py \
--model_name_or_path answerdotai/ModernBERT-large \
--dataset_name conll2003 \
--output_dir ./output \
--do_train \
--do_eval \
--do_predict \
--seed 1 \
--num_train_epochs 10 \
--eval_strategy epoch \
--save_strategy epoch \
--load_best_model_at_end \
--per_device_train_batch_size 4 \
--learning_rate 5e-06 \
--trust_remote_code=True
```

Unfortunately, I was only able to 95.20% on development set and 90.54% on test dataset.

Additionally, I performed fine-tuning with Flair library - that does not use the `ModernBertForTokenClassification` implementation and it also yielded about 95% on development set and ~90% on test dataset.

Now the question is: is there a kind of bug in the current released Transformers version (I tested with https://github.com/huggingface/transformers/commit/b5a557e5fe2d015bd36214a95878370eaed51571) or are there more tricks needed to get token classification working :thinking:

Many thanks in advance!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.