AnswerDotAI / AnswerDotAI/ModernBERT

Example of GreedyBestFitSequencePacker with HuggingFace Trainer

Open
#161 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Im experimenting with using ModernBERT as a cross encoder model. My training script is using the HF ModernBertForSequenceClassification and HF trainer. My dataset is long sequences (similar to MLDR) with a max input length of ~8k, but lots of variance in my sequence lengths. My training/inference should benefit greatly from the sequence packer outlined in the ModernBERT paper. Looking through the code I assume you use the `GreedyBestFitSequencePacker` for this. Is there an example you can share, similar to `finetune_modernbert_on_glue.ipynb`, that uses a the sequence packer instead of `DataCollatorWithPadding`?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.