Unable to reproduce Table 7 from the paper

Hi, I have been trying to replicate the BEIR scores in table 7 from the [paper](https://arxiv.org/pdf/2412.13663). I used the [train_st.py](https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/train_st.py) script as is, and trained on 2 GPUs for 1 epoch like this `accelerate launch --num_processes num_gpu train_st.py`, and then evaluated it on BEIR using MTEB library.

I use the hyper-parameters suggested in Table 9 (lr 8e-5 for ModernBert, 5e-5 for bert-base), rest default values from the script. I am not able to replicate the numbers, any idea what could be the difference? Could you please list the hyperparams you use? How many GPUs, batch size, or any special arguments to pass to MTEB?

Thanks!

Start with examples/train_st.py and the reported accelerate launch command, comparing its defaults with the hyperparameters in Table 9 of the linked paper. Check the MTEB evaluation setup and document the GPU count, batch size, special arguments, and other settings needed to reproduce the BEIR scores in Table 7.

Written by the indexing model from the issue text.