CPU running time of pretrained model RetinaFace-R50

I am running the pre-trained model RetinaFace-R50 and with my mac pro it takes about 4s (0.25fps) to process an image (only inference time, w/o loading/pre/post-processing).

The paper says that the bigger RetinaFace model, with ResNet152 backbone, runs in. 75 ms (13fps). This is more than 50 times faster than what I experience with a CPU and on a smaller model. Is this normal?

To draw an analogy, using the GPU on the MobileNet-0.25 model, the execution is about 12 times faster than with the CPU. However in my case I am using the CPU on a smaller model and, despite that, it is 50 times slower.

From the paper: