AnswerDotAI / AnswerDotAI/ModernBERT
On the performance of token classification
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Hi guys,
I spent some time evaluating ModernBERT Large on the standard CoNLL-2003 shared task.
For fine-tuning I used the following [code](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) to fine-tune:
```bash
python3 run_ner.py \
--model_name_or_path answerdotai/ModernBERT-large \
--dataset_name conll2003 \
--output_dir ./output \
--do_train \
--do_eval \
--do_predict \
--seed 1 \
--num_train_epochs 10 \
--eval_strategy epoch \
--save_strategy epoch \
--load_best_model_at_end \
--per_device_train_batch_size 4 \
--learning_rate 5e-06 \
--trust_remote_code=True
```
Unfortunately, I was only able to 95.20% on development set and 90.54% on test dataset.
Additionally, I performed fine-tuning with Flair library - that does not use the `ModernBertForTokenClassification` implementation and it also yielded about 95% on development set and ~90% on test dataset.
Now the question is: is there a kind of bug in the current released Transformers version (I tested with https://github.com/huggingface/transformers/commit/b5a557e5fe2d015bd36214a95878370eaed51571) or are there more tricks needed to get token classification working :thinking:
Many thanks in advance!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.