S2UT model pre-training using pretrained wav2vec 2 showing dimension mismatch

I am training S2UT models, for which two models are s2ut_transformer and s2ut_transformer_fisher.
Both are showing dimension mismatch with wav2vec2 base models; some other models show dict key mismatch,

Which wav2vec pre-trained model is compatible with the s2ut model?
Or suggest any available S2UT model along with multitask learning that can be given pretraining using wav2vec of some other relevant pre-trained models?

size mismatch for encoder.layer_norm.weight: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]).
size mismatch for encoder.layer_norm.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]).