problem about longformer inference of huggingface/transformers

when I use longformer in huggingface/transformers ,I find that the inference time of longformer-base-4096 is more than bert-base-cased.
However, the longformer should take less time than bert for long-document