Mismatch between pre-training and fine-tuning phase

As far as I'm aware, PRIMERA replaces sentences with \ tokens during pre-training. However, these \ tokens do not appear on fine-tuning phase, or when doing inference. Still, it was shown to achieve impressive results on Zero-Shot and Few-Shot Evaluation. I was wondering if PRIMERA had any strategies to reduce the mismatch between fine-tuning and pre-training phase ? (I did not see related information mentioned in the paper)