Trying to reproduce MJEPA-L on Audioset

Hi,

I'm trying to reproduce the base ViT-L results, and I wanted to ask about something I noticed during pre-training.

I followed the released config for the base ViT-L and trained for 600 epochs, with the same total batch size of 512:
https://github.com/facebookresearch/MJEPA/blob/main/configs/train/vitl16/pretrain-224px-16f.yaml

Near the end of training there is a jump in the loss. The paper reports 150k pre-training steps, which is epoch 500 (500 epochs × 300 ipe) and before this jump, so I used the epoch-500 checkpoint for evaluation. On the AudioSet-20K audio-video frozen evaluation I got around 18.7 mAP, compared to 42.90 in the paper.

Looking at the individual loss terms (plot attached below), the intra-modal losses (a2a, v2v, av2av) reach their lowest point around epoch 100 and then go back up before levelling off, while all the cross-modal losses (a2v, v2a, av2a, av2v, a2av, v2av) keep decreasing for the rest of training. This makes me suspect the representation may be collapsing after epoch 100. I also evaluated the epoch-100 checkpoint and got around 22 mAP, which is higher than epoch 500.

Can you heIp me resolve this and also I have a few questions
1. Did you observe similar behaviour in the individual losses during pre-training? If so, do you have any suggestions on how to avoid it, so the results are closer to the paper?
2. The config is set to 600 epochs (180k steps), but the paper reports 150k steps (epoch 500). Did you train for 600 epochs and use the epoch-500 checkpoint, or set training to 500 epochs and use the final checkpoint? If you trained for 600 epochs, did you also see the jump in the loss near the end?

3. Is there anything in my setup that I might have missed or done differently from your training?

Thank you

Start with configs/train/vitl16/pretrain-224px-16f.yaml and compare the reported 500-epoch checkpoint with the 600-epoch run and the epoch-100 checkpoint. Review the training and AudioSet-20K frozen evaluation setup against the issue's reported configuration; done means identifying the discrepancy or reproducing the paper's evaluation result.

Written by the indexing model from the issue text.