AnswerDotAI / AnswerDotAI/ModernBERT
Example of GreedyBestFitSequencePacker with HuggingFace Trainer
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Im experimenting with using ModernBERT as a cross encoder model. My training script is using the HF ModernBertForSequenceClassification and HF trainer. My dataset is long sequences (similar to MLDR) with a max input length of ~8k, but lots of variance in my sequence lengths. My training/inference should benefit greatly from the sequence packer outlined in the ModernBERT paper. Looking through the code I assume you use the `GreedyBestFitSequencePacker` for this. Is there an example you can share, similar to `finetune_modernbert_on_glue.ipynb`, that uses a the sequence packer instead of `DataCollatorWithPadding`?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.